diff --git a/docs/ABI.md b/docs/ABI.md new file mode 100644 index 0000000000000..af955437007fd --- /dev/null +++ b/docs/ABI.md @@ -0,0 +1,1176 @@ +The Swift ABI +============= + +Hard Constraints on Resilience +------------------------------ + +The root of a class hierarchy must remain stable, at pain of +invalidating the metaclass hierarchy. Note that a Swift class without an +explicit base class is implicitly rooted in the SwiftObject Objective-C +class. + +Type Layout +----------- + +### Fragile Struct and Tuple Layout + +Structs and tuples currently share the same layout algorithm, noted as +the "Universal" layout algorithm in the compiler implementation. The +algorithm is as follows: + +- Start with a **size** of **0** and an **alignment** of **1**. +- Iterate through the fields, in element order for tuples, or in `var` + declaration order for structs. For each field: + - Update **size** by rounding up to the **alignment of the + field**, that is, increasing it to the least value greater or + equal to **size** and evenly divisible by the **alignment of the + field**. + - Assign the **offset of the field** to the current value of + **size**. + - Update **size** by adding the **size of the field**. + - Update **alignment** to the max of **alignment** and the + **alignment of the field**. +- The final **size** and **alignment** are the size and alignment of + the aggregate. The **stride** of the type is the final **size** + rounded up to **alignment**. + +Note that this differs from C or LLVM's normal layout rules in that +*size* and *stride* are distinct; whereas C layout requires that an +embedded struct's size be padded out to its alignment and that nothing +be laid out there, Swift layout allows an outer struct to lay out fields +in the inner struct's tail padding, alignment permitting. Unlike C, +zero-sized structs and tuples are also allowed, and take up no storage +in enclosing aggregates. The Swift compiler emits LLVM packed struct +types with manual padding to get the necessary control over the binary +layout. Some examples: + + // LLVM <{ i64, i8 }> + struct S { + var x: Int + var y: UInt8 + } + + // LLVM <{ i8, [7 x i8], <{ i64, i8 }>, i8 }> + struct S2 { + var x: UInt8 + var s: S + var y: UInt8 + } + + // LLVM <{}> + struct Empty {} + + // LLVM <{ i64, i64 }> + struct ContainsEmpty { + var x: Int + var y: Empty + var z: Int + } + +### Class Layout + +Swift relies on the following assumptions about the Objective-C runtime, +which are therefore now part of the Objective-C ABI: + +- 32-bit platforms never have tagged pointers. ObjC pointer types are + either nil or an object pointer. +- On x86-64, a tagged pointer either sets the lowest bit of the + pointer or the highest bit of the pointer. Therefore, both of these + bits are zero if and only if the value is not a tagged pointer. +- On ARM64, a tagged pointer always sets the highest bit of + the pointer. +- 32-bit platforms never perform any isa masking. `object_getClass` is + always equivalent to `*(Class*)object`. +- 64-bit platforms perform isa masking only if the runtime exports a + symbol `uintptr_t objc_debug_isa_class_mask;`. If this symbol is + exported, `object_getClass` on a non-tagged pointer is always + equivalent to + `(Class)(objc_debug_isa_class_mask & *(uintptr_t*)object)`. +- The superclass field of a class object is always stored immediately + after the isa field. Its value is either nil or a pointer to the + class object for the superclass; it never has other bits set. + +The following assumptions are part of the Swift ABI: + +- Swift class pointers are never tagged pointers. + +TODO + +### Fragile Enum Layout + +In laying out enum types, the ABI attempts to avoid requiring additional +storage to store the tag for the enum case. The ABI chooses one of five +strategies based on the layout of the enum: + +#### Empty Enums + +In the degenerate case of an enum with no cases, the enum is an empty +type. + + enum Empty {} // => empty type + +#### Single-Case Enums + +In the degenerate case of an enum with a single case, there is no +discriminator needed, and the enum type has the exact same layout as its +case's data type, or is empty if the case has no data type. + + enum EmptyCase { case X } // => empty type + enum DataCase { case Y(Int, Double) } // => LLVM <{ i64, double }> + +#### C-Like Enums + +If none of the cases has a data type (a "C-like" enum), then the enum is +laid out as an integer tag with the minimal number of bits to contain +all of the cases. The machine-level layout of the type then follows +LLVM's data layout rules for integer types on the target platform. The +cases are assigned tag values in declaration order. + + enum EnumLike2 { // => LLVM i1 + case A // => i1 0 + case B // => i1 1 + } + + enum EnumLike8 { // => LLVM i3 + case A // => i3 0 + case B // => i3 1 + case C // => i3 2 + case D // etc. + case E + case F + case G + case H + } + +Discriminator values after the one used for the last case become *extra +inhabitants* of the enum type (see [Single-Payload Enums](#single-payload-enums)). + +#### Single-Payload Enums + +If an enum has a single case with a data type and one or more no-data +cases (a "single-payload" enum), then the case with data type is +represented using the data type's binary representation, with added zero +bits for tag if necessary. If the data type's binary representation has +**extra inhabitants**, that is, bit patterns with the size and alignment +of the type but which do not form valid values of that type, they are +used to represent the no-data cases, with extra inhabitants in order of +ascending numeric value matching no-data cases in declaration order. If +the type has *spare bits* (see [Multi-Payload Enums](#multi-payload-enums)), +they are used to form extra inhabitants. The enum value is then represented as +an integer with the storage size in bits of the data type. Extra inhabitants of +the payload type not used by the enum type become extra inhabitants of the enum +type itself. + + enum CharOrSectionMarker { => LLVM i32 + case Paragraph => i32 0x0020_0000 + case Char(UnicodeScalar) => i32 (zext i21 %Char to i32) + case Chapter => i32 0x0020_0001 + } + + CharOrSectionMarker.Char('\x00') => i32 0x0000_0000 + CharOrSectionMarker.Char('\u10FFFF') => i32 0x0010_FFFF + + enum CharOrSectionMarkerOrFootnoteMarker { => LLVM i32 + case CharOrSectionMarker(CharOrSectionMarker) => i32 %CharOrSectionMarker + case Asterisk => i32 0x0020_0002 + case Dagger => i32 0x0020_0003 + case DoubleDagger => i32 0x0020_0004 + } + +If the data type has no extra inhabitants, or there are not enough extra +inhabitants to represent all of the no-data cases, then a tag bit is +added to the enum's representation. The tag bit is set for the no-data +cases, which are then assigned values in the data area of the enum in +declaration order. + + enum IntOrInfinity { => LLVM <{ i64, i1 }> + case NegInfinity => <{ i64, i1 }> { 0, 1 } + case Int(Int) => <{ i64, i1 }> { %Int, 0 } + case PosInfinity => <{ i64, i1 }> { 1, 1 } + } + + IntOrInfinity.Int( 0) => <{ i64, i1 }> { 0, 0 } + IntOrInfinity.Int(20721) => <{ i64, i1 }> { 20721, 0 } + +#### Multi-Payload Enums + +If an enum has more than one case with data type, then a tag is +necessary to discriminate the data types. The ABI will first try to find +common **spare bits**, that is, bits in the data types' binary +representations which are either fixed-zero or ignored by valid values +of all of the data types. The tag will be scattered into these spare +bits as much as possible. Currently only spare bits of primitive integer +types, such as the high bits of an `i21` type, are considered. The enum +data is represented as an integer with the storage size in bits of the +largest data type. + + enum TerminalChar { => LLVM i32 + case Plain(UnicodeScalar) => i32 (zext i21 %Plain to i32) + case Bold(UnicodeScalar) => i32 (or (zext i21 %Bold to i32), 0x0020_0000) + case Underline(UnicodeScalar) => i32 (or (zext i21 %Underline to i32), 0x0040_0000) + case Blink(UnicodeScalar) => i32 (or (zext i21 %Blink to i32), 0x0060_0000) + case Empty => i32 0x0080_0000 + case Cursor => i32 0x0080_0001 + } + +If there are not enough spare bits to contain the tag, then additional +bits are added to the representation to contain the tag. Tag values are +assigned to data cases in declaration order. If there are no-data cases, +they are collected under a common tag, and assigned values in the data +area of the enum in declaration order. + + class Bignum {} + + enum IntDoubleOrBignum { => LLVM <{ i64, i2 }> + case Int(Int) => <{ i64, i2 }> { %Int, 0 } + case Double(Double) => <{ i64, i2 }> { (bitcast %Double to i64), 1 } + case Bignum(Bignum) => <{ i64, i2 }> { (ptrtoint %Bignum to i64), 2 } + } + +### Existential Container Layout + +Values of protocol type, protocol composition type, or "any" type +(`protocol<>`) are laid out using **existential containers** (so-called +because these types are "existential types" in type theory). + +#### Opaque Existential Containers + +If there is no class constraint on a protocol or protocol composition +type, the existential container has to accommodate a value of arbitrary +size and alignment. It does this using a **fixed-size buffer**, which is +three pointers in size and pointer-aligned. This either directly +contains the value, if its size and alignment are both less than or +equal to the fixed-size buffer's, or contains a pointer to a side +allocation owned by the existential container. The type of the contained +value is identified by its type metadata record, and witness tables for +all of the required protocol conformances are included. The layout is as +if declared in the following C struct: + + struct OpaqueExistentialContainer { + void *fixedSizeBuffer[3]; + Metadata *type; + WitnessTable *witnessTables[NUM_WITNESS_TABLES]; + }; + +#### Class Existential Containers + +If one or more of the protocols in a protocol or protocol composition +type have a class constraint, then only class values can be stored in +the existential container, and a more efficient representation is used. +Class instances are always a single pointer in size, so a fixed-size +buffer and potential side allocation is not needed, and class instances +always have a reference to their own type metadata, so the separate +metadata record is not needed. The layout is thus as if declared in the +following C struct: + + struct ClassExistentialContainer { + HeapObject *value; + WitnessTable *witnessTables[NUM_WITNESS_TABLES]; + }; + +Note that if no witness tables are needed, such as for the "any class" +type `protocol` or an Objective-C protocol type, then the only +element of the layout is the heap object pointer. This is ABI-compatible +with `id` and `id ` types in Objective-C. + +Type Metadata +------------- + +The Swift runtime keeps a **metadata record** for every type used in a +program, including every instantiation of generic types. These metadata +records can be used by (TODO: reflection and) debugger tools to discover +information about types. For non-generic nominal types, these metadata +records are generated statically by the compiler. For instances of +generic types, and for intrinsic types such as tuples, functions, +protocol compositions, etc., metadata records are lazily created by the +runtime as required. Every type has a unique metadata record; two +**metadata pointer** values are equal iff the types are equivalent. + +In the layout descriptions below, offsets are given relative to the +metadata pointer as an index into an array of pointers. On a 32-bit +platform, **offset 1** means an offset of 4 bytes, and on 64-bit +platforms, it means an offset of 8 bytes. + +### Common Metadata Layout + +All metadata records share a common header, with the following fields: + +- The **value witness table** pointer references a vtable of functions + that implement the value semantics of the type, providing + fundamental operations such as allocating, copying, and destroying + values of the type. The value witness table also records the size, + alignment, stride, and other fundamental properties of the type. The + value witness table pointer is at **offset -1** from the metadata + pointer, that is, the pointer-sized word **immediately before** the + pointer's referenced address. +- The **kind** field is a pointer-sized integer that describes the + kind of type the metadata describes. This field is at **offset 0** + from the metadata pointer. + + The current kind values are as follows: + + - [Struct metadata](#struct-metadata) has a kind of **1**. + - [Enum metadata](#enum-metadata) has a kind of **2**. + - **Opaque metadata** has a kind of **8**. This is used for + compiler `Builtin` primitives that have no additional + runtime information. + - [Tuple metadata](#tuple-metadata) has a kind of **9**. + - [Function metadata](#function-metadata) has a kind of **10**. + - [Protocol metadata](#protocol-metadata) has a kind of **12**. This is + used for protocol types, for protocol compositions, and for the "any" + type `protocol<>`. + - [Metatype metadata](#metatype-metadata) has a kind of **13**. + - [Class metadata](#class-metadata), instead of a kind, has an + *isa pointer* in its kind slot, pointing to the class's metaclass + record. This isa pointer is guaranteed to have an integer value larger + than **4096** and so can be discriminated from non-class kind values. + +### Struct Metadata + +In addition to the [common metadata layout](#common-metadata-layout) fields, +struct metadata records contain the following fields: + +- The [nominal type descriptor](#nominal-type-descriptor) is referenced at + **offset 1**. +- A reference to the **parent** metadata record is stored at **offset + 2**. For structs that are members of an enclosing nominal type, this + is a reference to the enclosing type's metadata. For top-level + structs, this is null. + + TODO: The parent pointer is currently always null. + +- A vector of **field offsets** begins at **offset 3**. For each field + of the struct, in `var` declaration order, the field's offset in + bytes from the beginning of the struct is stored as a + pointer-sized integer. +- If the struct is generic, then the [generic parameter + vector](#generic-parameter-vector) begins at **offset 3+n**, where **n** is + the number of fields in the struct. + +### Enum Metadata + +In addition to the [common metadata layout](#common-metadata-layout) fields, +enum metadata records contain the following fields: + +- The [nominal type descriptor](#nominal-type-descriptor) is referenced at + **offset 1**. +- A reference to the **parent** metadata record is stored at **offset + 2**. For enums that are members of an enclosing nominal type, this + is a reference to the enclosing type's metadata. For top-level + enums, this is null. + + TODO: The parent pointer is currently always null. + +- If the enum is generic, then the [generic parameter + vector](#generic-parameter-vector) begins at **offset 3**. + +### Tuple Metadata + +In addition to the [common metadata layout](#common-metadata-layout) fields, +tuple metadata records contain the following fields: + +- The **number of elements** in the tuple is a pointer-sized integer + at **offset 1**. +- The **labels string** is a pointer to a list of consecutive + null-terminated label names for the tuple at **offset 2**. Each + label name is given as a null-terminated, UTF-8-encoded string + in sequence. If the tuple has no labels, this is a null pointer. + + TODO: The labels string pointer is currently always null, and labels + are not factored into tuple metadata uniquing. + +- The **element vector** begins at **offset 3** and consists of a + vector of type–offset pairs. The metadata for the *n*th element's + type is a pointer at **offset 3+2\*n**. The offset in bytes from the + beginning of the tuple to the beginning of the *n*th element is at + **offset 3+2\*n+1**. + +### Function Metadata + +In addition to the [common metadata layout](#common-metadata-layout) fields, +function metadata records contain the following fields: + +- The number of arguments to the function is stored at **offset 1**. +- A reference to the **result type** metadata record is stored at + **offset 2**. If the function has multiple returns, this references + a [tuple metadata](#tuple-metadata) record. +- The **argument vector** begins at **offset 3** and consists of + pointers to metadata records of the function's arguments. + + If the function takes any **inout** arguments, a pointer to each + argument's metadata record will be appended separately, the lowest + bit being set if it is **inout**. Because of pointer alignment, the + lowest bit will always be free to hold this tag. + + If the function takes no **inout** arguments, there will be only one + pointer in the vector for the following cases: + + - 0 arguments: a [tuple metadata](#tuple-metadata) record for the empty + tuple + - 1 argument: the first and only argument's metadata record + - >1 argument: a [tuple metadata](#tuple-metadata) record containing + the arguments + +### Protocol Metadata + +In addition to the [common metadata layout](#common-metadata-layout) fields, +protocol metadata records contain the following fields: + +- A **layout flags** word is stored at **offset 1**. The bits of this + word describe the [existential container + layout](#existential-container-layout) used to represent values of the type. + The word is laid out as follows: + + - The **number of witness tables** is stored in the least + significant 31 bits. Values of the protocol type contain this + number of witness table pointers in their layout. + - The **class constraint** is stored at bit 31. This bit is set if + the type is **not** class-constrained, meaning that struct, + enum, or class values can be stored in the type. If not set, + then only class values can be stored in the type, and the type + uses a more efficient layout. + + Note that the field is pointer-sized, even though only the lowest 32 + bits are currently inhabited on all platforms. These values can be + derived from the [protocol descriptor](#protocol-descriptor) records, but are + pre-calculated for convenience. + +- The **number of protocols** that make up the protocol composition is + stored at **offset 2**. For the "any" types `protocol<>` or + `protocol`, this is zero. For a single-protocol type `P`, + this is one. For a protocol composition type `protocol`, + this is the number of protocols. +- The **protocol descriptor vector** begins at **offset 3**. This is + an inline array of pointers to the [protocol + descriptor](#protocol-descriptor) for every protocol in the composition, or + the single protocol descriptor for a protocol type. For an "any" type, there + is no protocol descriptor vector. + +### Metatype Metadata + +In addition to the [common metadata layout](#common-metadata-layout) fields, +metatype metadata records contain the following fields: + +- A reference to the metadata record for the **instance type** that + the metatype represents is stored at **offset 1**. + +### Class Metadata + +Class metadata is designed to interoperate with Objective-C; all class +metadata records are also valid Objective-C `Class` objects. Class +metadata pointers are used as the values of class metatypes, so a +derived class's metadata record also serves as a valid class metatype +value for all of its ancestor classes. + +- The **destructor pointer** is stored at **offset -2** from the + metadata pointer, behind the value witness table. This function is + invoked by Swift's deallocator when the class instance is destroyed. +- The **isa pointer** pointing to the class's Objective-C-compatible + metaclass record is stored at **offset 0**, in place of an integer + kind discriminator. +- The **super pointer** pointing to the metadata record for the + superclass is stored at **offset 1**. If the class is a root class, + it is null. +- Two words are reserved for use by the Objective-C runtime at + **offset 2** and **offset 3**. +- The **rodata pointer** is stored at **offset 4**; it points to an + Objective-C compatible rodata record for the class. This pointer + value includes a tag. The **low bit is always set to 1** for Swift + classes and always set to 0 for Objective-C classes. +- The **class flags** are a 32-bit field at **offset 5**. +- The **instance address point** is a 32-bit field following the + class flags. A pointer to an instance of this class points this + number of bytes after the beginning of the instance. +- The **instance size** is a 32-bit field following the instance + address point. This is the number of bytes of storage present in + every object of this type. +- The **instance alignment mask** is a 16-bit field following the + instance size. This is a set of low bits which must not be set in a + pointer to an instance of this class. +- The **runtime-reserved field** is a 16-bit field following the + instance alignment mask. The compiler initializes this to zero. +- The **class object size** is a 32-bit field following the + runtime-reserved field. This is the total number of bytes of storage + in the class metadata object. +- The **class object address point** is a 32-bit field following the + class object size. This is the number of bytes of storage in the + class metadata object. +- The [nominal type descriptor](#nominal-type-descriptor) for the most-derived + class type is referenced at an offset immediately following the class object + address point. This is **offset 8** on a 64-bit platform or **offset + 11** on a 32-bit platform. +- For each Swift class in the class's inheritance hierarchy, in order + starting from the root class and working down to the most derived + class, the following fields are present: + + - First, a reference to the **parent** metadata record is stored. + For classes that are members of an enclosing nominal type, this + is a reference to the enclosing type's metadata. For top-level + classes, this is null. + + TODO: The parent pointer is currently always null. + + - If the class is generic, its [generic parameter + vector](#generic-parameter-vector) is stored inline. + - The **vtable** is stored inline and contains a function pointer + to the implementation of every method of the class in + declaration order. + - If the layout of a class instance is dependent on its generic + parameters, then a **field offset vector** is stored inline, + containing offsets in bytes from an instance pointer to each + field of the class in declaration order. (For classes with fixed + layout, the field offsets are accessible statically from global + variables, similar to Objective-C ivar offsets.) + + Note that none of these fields are present for Objective-C base + classes in the inheritance hierarchy. + +### Generic Parameter Vector + +Metadata records for instances of generic types contain information +about their generic parameters. For each parameter of the type, a +reference to the metadata record for the type argument is stored. After +all of the type argument metadata references, for each type parameter, +if there are protocol requirements on that type parameter, a reference +to the witness table for each protocol it is required to conform to is +stored in declaration order. + +For example, given a generic type with the parameters ``, its +generic parameter record will consist of references to the metadata +records for `T`, `U`, and `V` in succession, as if laid out in a C +struct: + + struct GenericParameterVector { + TypeMetadata *T, *U, *V; + }; + +If we add protocol requirements to the parameters, for example, +`, V>`, then the type's +generic parameter vector contains witness tables for those protocols, as +if laid out: + + struct GenericParameterVector { + TypeMetadata *T, *U, *V; + RuncibleWitnessTable *T_Runcible; + FungibleWitnessTable *U_Fungible; + AnsibleWitnessTable *U_Ansible; + }; + +### Nominal Type Descriptor + +The metadata records for class, struct, and enum types contain a pointer +to a **nominal type descriptor**, which contains basic information about +the nominal type such as its name, members, and metadata layout. For a +generic type, one nominal type descriptor is shared for all +instantiations of the type. The layout is as follows: + +- The **kind** of type is stored at **offset 0**, which is as follows: + - **0** for a class, + - **1** for a struct, or + - **2** for an enum. +- The mangled **name** is referenced as a null-terminated C string at + **offset 1**. This name includes no bound generic parameters. +- The following four fields depend on the kind of nominal type. + - For a struct or class: + - The **number of fields** is stored at **offset 2**. This is + the length of the field offset vector in the metadata + record, if any. + - The **offset to the field offset vector** is stored at + **offset 3**. This is the offset in pointer-sized words of + the field offset vector for the type in the metadata record. + If no field offset vector is stored in the metadata record, + this is zero. + - The **field names** are referenced as a + doubly-null-terminated list of C strings at **offset 4**. + The order of names corresponds to the order of fields in the + field offset vector. + - The **field type accessor** is a function pointer at + **offset 5**. If non-null, the function takes a pointer to + an instance of type metadata for the nominal type, and + returns a pointer to an array of type metadata references + for the types of the fields of that instance. The order + matches that of the field offset vector and field name list. + - For an enum: + - The **number of payload cases** and **payload size offset** + are stored at **offset 2**. The least significant 24 bits + are the number of payload cases, and the most significant 8 + bits are the offset of the payload size in the type + metadata, if present. + - The **number of no-payload cases** is stored at **offset + 3**. + - The **case names** are referenced as a + doubly-null-terminated list of C strings at **offset 4**. + The names are ordered such that payload cases come first, + followed by no-payload cases. Within each half of the list, + the order of names corresponds to the order of cases in the + enum declaration. + - The **case type accessor** is a function pointer at **offset + 5**. If non-null, the function takes a pointer to an + instance of type metadata for the enum, and returns a + pointer to an array of type metadata references for the + types of the cases of that instance. The order matches that + of the case name list. This function is similar to the field + type accessor for a struct, except also the least + significant bit of each element in the result is set if the + enum case is an **indirect case**. +- If the nominal type is generic, a pointer to the **metadata + pattern** that is used to form instances of the type is stored at + **offset 6**. The pointer is null if the type is not generic. +- The **generic parameter descriptor** begins at **offset 7**. This + describes the layout of the generic parameter vector in the metadata + record: + - The **offset of the generic parameter vector** is stored at + **offset 7**. This is the offset in pointer-sized words of the + generic parameter vector inside the metadata record. If the type + is not generic, this is zero. + - The **number of type parameters** is stored at **offset 8**. + This count includes associated types of type parameters with + protocol constraints. + - The **number of type parameters** is stored at **offset 9**. + This count includes only the primary formal type parameters. + - For each type parameter **n**, the following fields are stored: + - The **number of witnesses** for the type parameter is stored + at **offset 10+n**. This is the number of witness table + pointers that are stored for the type parameter in the + generic parameter vector. + +Note that there is no nominal type descriptor for protocols or protocol +types. See the [protocol descriptor](#protocol-descriptor) description below. + +### Protocol Descriptor + +Protocol metadata contains references to zero, one, or more **protocol +descriptors** that describe the protocols values of the type are +required to conform to. The protocol descriptor is laid out to be +compatible with Objective-C `Protocol` objects. The layout is as +follows: + +- An **isa** placeholder is stored at **offset 0**. This field is + populated by the Objective-C runtime. +- The mangled **name** is referenced as a null-terminated C string at + **offset 1**. +- If the protocol inherits one or more other protocols, a pointer to + the **inherited protocols list** is stored at **offset 2**. The list + starts with the number of inherited protocols as a pointer-sized + integer, and is followed by that many protocol descriptor pointers. + If the protocol inherits no other protocols, this pointer is null. +- For an ObjC-compatible protocol, its **required instance methods** + are stored at **offset 3** as an ObjC-compatible method list. This + is null for native Swift protocols. +- For an ObjC-compatible protocol, its **required class methods** are + stored at **offset 4** as an ObjC-compatible method list. This is + null for native Swift protocols. +- For an ObjC-compatible protocol, its **optional instance methods** + are stored at **offset 5** as an ObjC-compatible method list. This + is null for native Swift protocols. +- For an ObjC-compatible protocol, its **optional class methods** are + stored at **offset 6** as an ObjC-compatible method list. This is + null for native Swift protocols. +- For an ObjC-compatible protocol, its **instance properties** are + stored at **offset 7** as an ObjC-compatible property list. This is + null for native Swift protocols. +- The **size** of the protocol descriptor record is stored as a 32-bit + integer at **offset 8**. This is currently 72 on 64-bit platforms + and 40 on 32-bit platforms. +- **Flags** are stored as a 32-bit integer after the size. The + following bits are currently used (counting from least significant + bit zero): + - **Bit 0** is the **Swift bit**. It is set for all protocols + defined in Swift and unset for protocols defined in Objective-C. + - **Bit 1** is the **class constraint bit**. It is set if the + protocol is **not** class-constrained, meaning that any struct, + enum, or class type may conform to the protocol. It is unset if + only classes can conform to the protocol. (The inverted meaning + is for compatibility with Objective-C protocol records, in which + the bit is never set. Objective-C protocols can only be + conformed to by classes.) + - **Bit 2** is the **witness table bit**. It is set if dispatch to + the protocol's methods is done through a witness table, which is + either passed as an extra parameter to generic functions or + included in the [existential + container layout](#existential-container-layout) of protocol types. It + is unset if dispatch is done through `objc_msgSend` and requires no + additional information to accompany a value of conforming type. + - **Bit 31** is set by the Objective-C runtime when it has done + its initialization of the protocol record. It is unused by the + Swift runtime. + +Heap Objects +------------ + +### Heap Metadata + +### Heap Object Header + +Mangling +-------- + + mangled-name ::= '_T' global + +All Swift-mangled names begin with this prefix. + +### Globals + + global ::= 't' type // standalone type (for DWARF) + global ::= 'M' type // type metadata (address point) + // -- type starts with [BCOSTV] + global ::= 'Mf' type // 'full' type metadata (start of object) + global ::= 'MP' type // type metadata pattern + global ::= 'Ma' type // type metadata access function + global ::= 'ML' type // type metadata lazy cache variable + global ::= 'Mm' type // class metaclass + global ::= 'Mn' nominal-type // nominal type descriptor + global ::= 'Mp' protocol // protocol descriptor + global ::= 'PA' .* // partial application forwarder + global ::= 'PAo' .* // ObjC partial application forwarder + global ::= 'w' value-witness-kind type // value witness + global ::= 'WV' type // value witness table + global ::= 'Wo' entity // witness table offset + global ::= 'Wv' directness entity // field offset + global ::= 'WP' protocol-conformance // protocol witness table + global ::= 'Wa' protocol-conformance // protocol witness table accessor + global ::= 'Wl' type protocol-conformance // lazy protocol witness table accessor + global ::= 'WL' protocol-conformance // lazy protocol witness table cache variable + global ::= 'WD' protocol-conformance // dependent proto witness table generator + global ::= 'Wd' protocol-conformance // dependent proto witness table template + global ::= entity // some identifiable thing + global ::= 'TO' global // ObjC-as-swift thunk + global ::= 'To' global // swift-as-ObjC thunk + global ::= 'TD' global // dynamic dispatch thunk + global ::= 'Td' global // direct method reference thunk + global ::= 'TR' reabstract-signature // reabstraction thunk helper function + global ::= 'Tr' reabstract-signature // reabstraction thunk + + global ::= 'TS' specializationinfo '_' mangled-name + specializationinfo ::= 'g' passid (type protocol-conformance* '_')+ // Generic specialization info. + specializationinfo ::= 'f' passid (funcspecializationarginfo '_')+ // Function signature specialization kind + passid ::= integer // The id of the pass that generated this specialization. + funcsigspecializationarginfo ::= 'cl' closurename type* // Closure specialized with closed over types in argument order. + funcsigspecializationarginfo ::= 'n' // Unmodified argument + funcsigspecializationarginfo ::= 'cp' funcsigspecializationconstantproppayload // Constant propagated argument + funcsigspecializationarginfo ::= 'd' // Dead argument + funcsigspecializationarginfo ::= 'g' 's'? // Owned => Guaranteed and Exploded if 's' present. + funcsigspecializationarginfo ::= 's' // Exploded + funcsigspecializationconstantpropinfo ::= 'fr' mangled-name + funcsigspecializationconstantpropinfo ::= 'g' mangled-name + funcsigspecializationconstantpropinfo ::= 'i' 64-bit-integer + funcsigspecializationconstantpropinfo ::= 'fl' float-as-64-bit-integer + funcsigspecializationconstantpropinfo ::= 'se' stringencoding 'v' md5hash + + global ::= 'TV' global // vtable override thunk + global ::= 'TW' protocol-conformance entity + // protocol witness thunk + entity ::= nominal-type // named type declaration + entity ::= static? entity-kind context entity-name + entity-kind ::= 'F' // function (ctor, accessor, etc.) + entity-kind ::= 'v' // variable (let/var) + entity-kind ::= 'i' // subscript ('i'ndex) itself (not the individual accessors) + entity-kind ::= 'I' // initializer + entity-name ::= decl-name type // named declaration + entity-name ::= 'A' index // default argument generator + entity-name ::= 'a' addressor-kind decl-name type // mutable addressor + entity-name ::= 'C' type // allocating constructor + entity-name ::= 'c' type // non-allocating constructor + entity-name ::= 'D' // deallocating destructor; untyped + entity-name ::= 'd' // non-deallocating destructor; untyped + entity-name ::= 'g' decl-name type // getter + entity-name ::= 'i' // non-local variable initializer + entity-name ::= 'l' addressor-kind decl-name type // non-mutable addressor + entity-name ::= 'm' decl-name type // materializeForSet + entity-name ::= 's' decl-name type // setter + entity-name ::= 'U' index type // explicit anonymous closure expression + entity-name ::= 'u' index type // implicit anonymous closure + entity-name ::= 'w' decl-name type // willSet + entity-name ::= 'W' decl-name type // didSet + static ::= 'Z' // entity is a static member of a type + decl-name ::= identifier + decl-name ::= local-decl-name + decl-name ::= private-decl-name + local-decl-name ::= 'L' index identifier // locally-discriminated declaration + private-decl-name ::= 'P' identifier identifier // file-discriminated declaration + reabstract-signature ::= ('G' generic-signature)? type type + addressor-kind ::= 'u' // unsafe addressor (no owner) + addressor-kind ::= 'O' // owning addressor (non-native owner) + addressor-kind ::= 'o' // owning addressor (native owner) + addressor-kind ::= 'p' // pinning addressor (native owner) + +An `entity` starts with a `nominal-type-kind` (`[COPV]`), a substitution +(`[Ss]`) of a nominal type, or an `entity-kind` (`[FIiv]`). + +An `entity-name` starts with `[AaCcDggis]` or a `decl-name`. A +`decl-name` starts with `[LP]` or an `identifier` (`[0-9oX]`). + +A `context` starts with either an `entity`, an `extension` (which starts +with `[Ee]`), or a `module`, which might be an `identifier` (`[0-9oX]`) +or a substitution of a module (`[Ss]`). + +A global mangling starts with an `entity` or `[MTWw]`. + +If a partial application forwarder is for a static symbol, its name will +start with the sequence `_TPA_` followed by the mangled symbol name of +the forwarder's destination. + +A generic specialization mangling consists of a header, specifying the +types and conformances used to specialize the generic function, followed +by the full mangled name of the original unspecialized generic symbol. + +The first identifier in a `` is a string that +represents the file the original declaration came from. It should be +considered unique within the enclosing module. The second identifier is +the name of the entity. + +Not all declarations marked `private` declarations will use the +`` mangling; if the entity's context is enough to +uniquely identify the entity, the simple `identifier` form is preferred. + +The types in a `` are always non-polymorphic +`` types. + +### Direct and Indirect Symbols + + directness ::= 'd' // direct + directness ::= 'i' // indirect + +A direct symbol resolves directly to the address of an object. An +indirect symbol resolves to the address of a pointer to the object. They +are distinct manglings to make a certain class of bugs immediately +obvious. + +The terminology is slightly overloaded when discussing offsets. A direct +offset resolves to a variable holding the true offset. An indirect +offset resolves to a variable holding an offset to be applied to type +metadata to get the address of the true offset. (Offset variables are +required when the object being accessed lies within a resilient +structure. When the layout of the object may depend on generic +arguments, these offsets must be kept in metadata. Indirect field +offsets are therefore required when accessing fields in generic types +where the metadata itself has unknown layout.) + +### Declaration Contexts + + context ::= module + context ::= extension + context ::= entity + module ::= substitution // other substitution + module ::= identifier // module name + module ::= known-module // abbreviation + extension ::= 'E' module entity + extension ::= 'e' module generic-signature entity + +These manglings identify the enclosing context in which an entity was +declared, such as its enclosing module, function, or nominal type. + +An `extension` mangling is used whenever an entity's declaration context +is an extension *and* the entity being extended is in a different +module. In this case the extension's module is mangled first, followed +by the entity being extended. If the extension and the extended entity +are in the same module, the plain `entity` mangling is preferred. If the +extension is constrained, the constraints on the extension are mangled +in its generic signature. + +When mangling the context of a local entity within a constructor or +destructor, the non-allocating or non-deallocating variant is used. + +### Types + + type ::= 'Bb' // Builtin.BridgeObject + type ::= 'BB' // Builtin.UnsafeValueBuffer + type ::= 'Bf' natural '_' // Builtin.Float + type ::= 'Bi' natural '_' // Builtin.Int + type ::= 'BO' // Builtin.ObjCPointer + type ::= 'Bo' // Builtin.ObjectPointer + type ::= 'Bp' // Builtin.RawPointer + type ::= 'Bv' natural type // Builtin.Vecx + type ::= 'Bw' // Builtin.Word + type ::= nominal-type + type ::= associated-type + type ::= 'a' context identifier // Type alias (DWARF only) + type ::= 'b' type type // objc block function type + type ::= 'c' type type // C function pointer type + type ::= 'F' throws-annotation? type type // function type + type ::= 'f' throws-annotation? type type // uncurried function type + type ::= 'G' type + '_' // generic type application + type ::= 'K' type type // @auto_closure function type + type ::= 'M' type // metatype without representation + type ::= 'XM' metatype-repr type // metatype with representation + type ::= 'P' protocol-list '_' // protocol type + type ::= 'PM' type // existential metatype without representation + type ::= 'XPM' metatype-repr type // existential metatype with representation + type ::= archetype + type ::= 'R' type // inout + type ::= 'T' tuple-element* '_' // tuple + type ::= 't' tuple-element* '_' // variadic tuple + type ::= 'Xo' type // @unowned type + type ::= 'Xu' type // @unowned(unsafe) type + type ::= 'Xw' type // @weak type + type ::= 'XF' impl-function-type // function implementation type + type ::= 'Xf' type type // @thin function type + nominal-type ::= known-nominal-type + nominal-type ::= substitution + nominal-type ::= nominal-type-kind declaration-name + nominal-type-kind ::= 'C' // class + nominal-type-kind ::= 'O' // enum + nominal-type-kind ::= 'V' // struct + archetype ::= 'Q' index // archetype with depth=0, idx=N + archetype ::= 'Qd' index index // archetype with depth=M+1, idx=N + archetype ::= associated-type + archetype ::= qualified-archetype + associated-type ::= substitution + associated-type ::= 'Q' protocol-context // self type of protocol + associated-type ::= 'Q' archetype identifier // associated type + qualified-archetype ::= 'Qq' index context // archetype+context (DWARF only) + protocol-context ::= 'P' protocol + tuple-element ::= identifier? type + metatype-repr ::= 't' // Thin metatype representation + metatype-repr ::= 'T' // Thick metatype representation + metatype-repr ::= 'o' // ObjC metatype representation + throws-annotation ::= 'z' // 'throws' annotation on function types + + + type ::= 'u' generic-signature type // generic type + type ::= 'x' // generic param, depth=0, idx=0 + type ::= 'q' generic-param-index // dependent generic parameter + type ::= 'q' type assoc-type-name // associated type of non-generic param + type ::= 'w' generic-param-index assoc-type-name // associated type + type ::= 'W' generic-param-index assoc-type-name+ '_' // associated type at depth + + generic-param-index ::= 'x' // depth = 0, idx = 0 + generic-param-index ::= index // depth = 0, idx = N+1 + generic-param-index ::= 'd' index index // depth = M+1, idx = N + +`` never begins or ends with a number. `` never begins with +an underscore. `` never begins with `d`. `` never begins +with `z`. + +Note that protocols mangle differently as types and as contexts. A +protocol context always consists of a single protocol name and so +mangles without a trailing underscore. A protocol type can have zero, +one, or many protocol bounds which are juxtaposed and terminated with a +trailing underscore. + + assoc-type-name ::= ('P' protocol-name)? identifier + assoc-type-name ::= substitution + +Associated types use an abbreviated mangling when the base generic +parameter or associated type is constrained by a single protocol +requirement. The associated type in this case can be referenced +unambiguously by name alone. If the base has multiple conformance +constraints, then the protocol name is mangled in to disambiguate. + + impl-function-type ::= + impl-callee-convention impl-function-attribute* generic-signature? '_' + impl-parameter* '_' impl-result* '_' + impl-callee-convention ::= 't' // thin + impl-callee-convention ::= impl-convention // thick, callee transferred with given convention + impl-convention ::= 'a' // direct, autoreleased + impl-convention ::= 'd' // direct, no ownership transfer + impl-convention ::= 'D' // direct, no ownership transfer, + // dependent on 'self' parameter + impl-convention ::= 'g' // direct, guaranteed + impl-convention ::= 'e' // direct, deallocating + impl-convention ::= 'i' // indirect, ownership transfer + impl-convention ::= 'l' // indirect, inout + impl-convention ::= 'G' // indirect, guaranteed + impl-convention ::= 'o' // direct, ownership transfer + impl-convention ::= 'z' impl-convention // error result + impl-function-attribute ::= 'Cb' // compatible with C block invocation function + impl-function-attribute ::= 'Cc' // compatible with C global function + impl-function-attribute ::= 'Cm' // compatible with Swift method + impl-function-attribute ::= 'CO' // compatible with ObjC method + impl-function-attribute ::= 'Cw' // compatible with protocol witness + impl-function-attribute ::= 'N' // noreturn + impl-function-attribute ::= 'G' // generic + impl-parameter ::= impl-convention type + impl-result ::= impl-convention type + +For the most part, manglings follow the structure of formal language +types. However, in some cases it is more useful to encode the exact +implementation details of a function type. + +Any `` productions must appear in the order in +which they are specified above: e.g. a noreturn C function is mangled +with `CcN`. + +Note that the convention and function-attribute productions do not need +to be disambiguated from the start of a ``. + +### Generics + + protocol-conformance ::= ('u' generic-signature)? type protocol module + +`` refers to a type's conformance to a protocol. +The named module is the one containing the extension or type declaration +that declared the conformance. + + generic-signature ::= (generic-param-count+)? ('R' requirement*)? 'r' + generic-param-count ::= 'z' // zero parameters + generic-param-count ::= index // N+1 parameters + requirement ::= type-param protocol-name // protocol requirement + requirement ::= type-param type // base class requirement + // type starts with [CS] + requirement ::= type-param 'z' type // 'z'ame-type requirement + + // Special type mangling for type params that saves the initial 'q' on + // generic params + type-param ::= generic-param-index // generic parameter + type-param ::= 'w' generic-param-index assoc-type-name // associated type + type-param ::= 'W' generic-param-index assoc-type-name+ '_' + +A generic signature begins by describing the number of generic +parameters at each depth of the signature, followed by the requirements. +As a special case, no `generic-param-count` values indicates a single +generic parameter at the outermost depth: + + urFq_q_ // T_0_0 -> T_0_0 + u_0_rFq_qd_0_ // T_0_0 -> T_1_1 + +### Value Witnesses + +TODO: document these + + value-witness-kind ::= 'al' // allocateBuffer + value-witness-kind ::= 'ca' // assignWithCopy + value-witness-kind ::= 'ta' // assignWithTake + value-witness-kind ::= 'de' // deallocateBuffer + value-witness-kind ::= 'xx' // destroy + value-witness-kind ::= 'XX' // destroyBuffer + value-witness-kind ::= 'Xx' // destroyArray + value-witness-kind ::= 'CP' // initializeBufferWithCopyOfBuffer + value-witness-kind ::= 'Cp' // initializeBufferWithCopy + value-witness-kind ::= 'cp' // initializeWithCopy + value-witness-kind ::= 'TK' // initializeBufferWithTakeOfBuffer + value-witness-kind ::= 'Tk' // initializeBufferWithTake + value-witness-kind ::= 'tk' // initializeWithTake + value-witness-kind ::= 'pr' // projectBuffer + value-witness-kind ::= 'xs' // storeExtraInhabitant + value-witness-kind ::= 'xg' // getExtraInhabitantIndex + value-witness-kind ::= 'Cc' // initializeArrayWithCopy + value-witness-kind ::= 'Tt' // initializeArrayWithTakeFrontToBack + value-witness-kind ::= 'tT' // initializeArrayWithTakeBackToFront + value-witness-kind ::= 'ug' // getEnumTag + value-witness-kind ::= 'up' // destructiveProjectEnumData + +`` differentiates the kinds of value witness +functions for a type. + +### Identifiers + + identifier ::= natural identifier-start-char identifier-char* + identifier ::= 'o' operator-fixity natural operator-char+ + + operator-fixity ::= 'p' // prefix operator + operator-fixity ::= 'P' // postfix operator + operator-fixity ::= 'i' // infix operator + + operator-char ::= 'a' // & 'and' + operator-char ::= 'c' // @ 'commercial at' + operator-char ::= 'd' // / 'divide' + operator-char ::= 'e' // = 'equals' + operator-char ::= 'g' // > 'greater' + operator-char ::= 'l' // < 'less' + operator-char ::= 'm' // * 'multiply' + operator-char ::= 'n' // ! 'not' + operator-char ::= 'o' // | 'or' + operator-char ::= 'p' // + 'plus' + operator-char ::= 'q' // ? 'question' + operator-char ::= 'r' // % 'remainder' + operator-char ::= 's' // - 'subtract' + operator-char ::= 't' // ~ 'tilde' + operator-char ::= 'x' // ^ 'xor' + operator-char ::= 'z' // . 'zperiod' + +`` is run-length encoded: the natural indicates how many +characters follow. Operator characters are mapped to letter characters +as given. In neither case can an identifier start with a digit, so +there's no ambiguity with the run-length. + + identifier ::= 'X' natural identifier-start-char identifier-char* + identifier ::= 'X' 'o' operator-fixity natural identifier-char* + +Identifiers that contain non-ASCII characters are encoded using the +Punycode algorithm specified in RFC 3492, with the modifications that +`_` is used as the encoding delimiter, and uppercase letters A through J +are used in place of digits 0 through 9 in the encoding character set. +The mangling then consists of an `X` followed by the run length of the +encoded string and the encoded string itself. For example, the +identifier `vergüenza` is mangled to `X12vergenza_JFa`. (The encoding in +standard Punycode would be `vergenza-95a`) + +Operators that contain non-ASCII characters are mangled by first mapping +the ASCII operator characters to letters as for pure ASCII operator +names, then Punycode-encoding the substituted string. The mangling then +consists of `Xo` followed by the fixity, run length of the encoded +string, and the encoded string itself. For example, the infix operator +`«+»` is mangled to `Xoi7p_qcaDc` (`p_qcaDc` being the encoding of the +substituted string `«p»`). + +### Substitutions + + substitution ::= 'S' index + +`` is a back-reference to a previously mangled entity. The +mangling algorithm maintains a mapping of entities to substitution +indices as it runs. When an entity that can be represented by a +substitution (a module, nominal type, or protocol) is mangled, a +substitution is first looked for in the substitution map, and if it is +present, the entity is mangled using the associated substitution index. +Otherwise, the entity is mangled normally, and it is then added to the +substitution map and associated with the next available substitution +index. + +For example, in mangling a function type +`(zim.zang.zung, zim.zang.zung, zim.zippity) -> zim.zang.zoo` (with +module `zim` and class `zim.zang`), the recurring contexts `zim`, +`zim.zang`, and `zim.zang.zung` will be mangled using substitutions +after being mangled for the first time. The first argument type will +mangle in long form, `CC3zim4zang4zung`, and in doing so, `zim` will +acquire substitution `S_`, `zim.zang` will acquire substitution `S0_`, +and `zim.zang.zung` will acquire `S1_`. The second argument is the same +as the first and will mangle using its substitution, `CS1_`. The third +argument type will mangle using the substitution for `zim`, +`CS_7zippity`. (It also acquires substitution `S2_` which would be used +if it mangled again.) The result type will mangle using the substitution +for `zim.zang`, `CS0_zoo` (and acquire substitution `S3_`). The full +function type thus mangles as +`fTCC3zim4zang4zungCS1_CS_7zippity_CS0_zoo`. + + substitution ::= 's' + +The special substitution `s` is used for the `Swift` standard library +module. + +### Predefined Substitutions + + known-module ::= 's' // Swift + known-module ::= 'SC' // C + known-module ::= 'So' // Objective-C + known-nominal-type ::= 'Sa' // Swift.Array + known-nominal-type ::= 'Sb' // Swift.Bool + known-nominal-type ::= 'Sc' // Swift.UnicodeScalar + known-nominal-type ::= 'Sd' // Swift.Float64 + known-nominal-type ::= 'Sf' // Swift.Float32 + known-nominal-type ::= 'Si' // Swift.Int + known-nominal-type ::= 'SP' // Swift.UnsafePointer + known-nominal-type ::= 'Sp' // Swift.UnsafeMutablePointer + known-nominal-type ::= 'SQ' // Swift.ImplicitlyUnwrappedOptional + known-nominal-type ::= 'Sq' // Swift.Optional + known-nominal-type ::= 'SR' // Swift.UnsafeBufferPointer + known-nominal-type ::= 'Sr' // Swift.UnsafeMutableBufferPointer + known-nominal-type ::= 'SS' // Swift.String + known-nominal-type ::= 'Su' // Swift.UInt + +`` and `` are built-in substitutions +for certain common entities. Like any other substitution, they all start +with 'S'. + +The Objective-C module is used as the context for mangling Objective-C +classes as ``s. + +### Indexes + + index ::= '_' // 0 + index ::= natural '_' // N+1 + natural ::= [0-9]+ + +`` is a production for encoding numbers in contexts that can't +end in a digit; it's optimized for encoding smaller numbers. diff --git a/docs/ABI.rst b/docs/ABI.rst deleted file mode 100644 index 30f570f91dd32..0000000000000 --- a/docs/ABI.rst +++ /dev/null @@ -1,1230 +0,0 @@ -:orphan: - -.. @raise litre.TestsAreMissing -.. _ABI: - -The Swift ABI -============= - -.. contents:: - -Hard Constraints on Resilience ------------------------------- - -The root of a class hierarchy must remain stable, at pain of -invalidating the metaclass hierarchy. Note that a Swift class without an -explicit base class is implicitly rooted in the SwiftObject -Objective-C class. - -Type Layout ------------ - -Fragile Struct and Tuple Layout -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Structs and tuples currently share the same layout algorithm, noted as the -"Universal" layout algorithm in the compiler implementation. The algorithm -is as follows: - -- Start with a **size** of **0** and an **alignment** of **1**. -- Iterate through the fields, in element order for tuples, or in ``var`` - declaration order for structs. For each field: - - * Update **size** by rounding up to the **alignment of the field**, that is, - increasing it to the least value greater or equal to **size** and evenly - divisible by the **alignment of the field**. - * Assign the **offset of the field** to the current value of **size**. - * Update **size** by adding the **size of the field**. - * Update **alignment** to the max of **alignment** and the - **alignment of the field**. - -- The final **size** and **alignment** are the size and alignment of the - aggregate. The **stride** of the type is the final **size** rounded up to - **alignment**. - -Note that this differs from C or LLVM's normal layout rules in that *size* -and *stride* are distinct; whereas C layout requires that an embedded struct's -size be padded out to its alignment and that nothing be laid out there, -Swift layout allows an outer struct to lay out fields in the inner struct's -tail padding, alignment permitting. Unlike C, zero-sized structs and tuples -are also allowed, and take up no storage in enclosing aggregates. The Swift -compiler emits LLVM packed struct types with manual padding to get the -necessary control over the binary layout. Some examples: - -:: - - // LLVM <{ i64, i8 }> - struct S { - var x: Int - var y: UInt8 - } - - // LLVM <{ i8, [7 x i8], <{ i64, i8 }>, i8 }> - struct S2 { - var x: UInt8 - var s: S - var y: UInt8 - } - - // LLVM <{}> - struct Empty {} - - // LLVM <{ i64, i64 }> - struct ContainsEmpty { - var x: Int - var y: Empty - var z: Int - } - -Class Layout -~~~~~~~~~~~~ - -Swift relies on the following assumptions about the Objective-C runtime, -which are therefore now part of the Objective-C ABI: - -- 32-bit platforms never have tagged pointers. ObjC pointer types are - either nil or an object pointer. - -- On x86-64, a tagged pointer either sets the lowest bit of the pointer - or the highest bit of the pointer. Therefore, both of these bits are - zero if and only if the value is not a tagged pointer. - -- On ARM64, a tagged pointer always sets the highest bit of the pointer. - -- 32-bit platforms never perform any isa masking. ``object_getClass`` - is always equivalent to ``*(Class*)object``. - -- 64-bit platforms perform isa masking only if the runtime exports a - symbol ``uintptr_t objc_debug_isa_class_mask;``. If this symbol - is exported, ``object_getClass`` on a non-tagged pointer is always - equivalent to ``(Class)(objc_debug_isa_class_mask & *(uintptr_t*)object)``. - -- The superclass field of a class object is always stored immediately - after the isa field. Its value is either nil or a pointer to the - class object for the superclass; it never has other bits set. - -The following assumptions are part of the Swift ABI: - -- Swift class pointers are never tagged pointers. - -TODO - -Fragile Enum Layout -~~~~~~~~~~~~~~~~~~~ - -In laying out enum types, the ABI attempts to avoid requiring additional -storage to store the tag for the enum case. The ABI chooses one of five -strategies based on the layout of the enum: - -Empty Enums -``````````` - -In the degenerate case of an enum with no cases, the enum is an empty type. - -:: - - enum Empty {} // => empty type - -Single-Case Enums -````````````````` - -In the degenerate case of an enum with a single case, there is no -discriminator needed, and the enum type has the exact same layout as its -case's data type, or is empty if the case has no data type. - -:: - - enum EmptyCase { case X } // => empty type - enum DataCase { case Y(Int, Double) } // => LLVM <{ i64, double }> - -C-Like Enums -```````````` - -If none of the cases has a data type (a "C-like" enum), then the enum -is laid out as an integer tag with the minimal number of bits to contain -all of the cases. The machine-level layout of the type then follows LLVM's -data layout rules for integer types on the target platform. The cases are -assigned tag values in declaration order. - -:: - - enum EnumLike2 { // => LLVM i1 - case A // => i1 0 - case B // => i1 1 - } - - enum EnumLike8 { // => LLVM i3 - case A // => i3 0 - case B // => i3 1 - case C // => i3 2 - case D // etc. - case E - case F - case G - case H - } - -Discriminator values after the one used for the last case become *extra -inhabitants* of the enum type (see `Single-Payload Enums`_). - -Single-Payload Enums -```````````````````` - -If an enum has a single case with a data type and one or more no-data cases -(a "single-payload" enum), then the case with data type is represented using -the data type's binary representation, with added zero bits for tag if -necessary. If the data type's binary representation -has **extra inhabitants**, that is, bit patterns with the size and alignment of -the type but which do not form valid values of that type, they are used to -represent the no-data cases, with extra inhabitants in order of ascending -numeric value matching no-data cases in declaration order. If the type -has *spare bits* (see `Multi-Payload Enums`_), they are used to form extra -inhabitants. The enum value is then represented as an integer with the storage -size in bits of the data type. Extra inhabitants of the payload type not used -by the enum type become extra inhabitants of the enum type itself. - -:: - - enum CharOrSectionMarker { => LLVM i32 - case Paragraph => i32 0x0020_0000 - case Char(UnicodeScalar) => i32 (zext i21 %Char to i32) - case Chapter => i32 0x0020_0001 - } - - CharOrSectionMarker.Char('\x00') => i32 0x0000_0000 - CharOrSectionMarker.Char('\u10FFFF') => i32 0x0010_FFFF - - enum CharOrSectionMarkerOrFootnoteMarker { => LLVM i32 - case CharOrSectionMarker(CharOrSectionMarker) => i32 %CharOrSectionMarker - case Asterisk => i32 0x0020_0002 - case Dagger => i32 0x0020_0003 - case DoubleDagger => i32 0x0020_0004 - } - -If the data type has no extra inhabitants, or there are not enough extra -inhabitants to represent all of the no-data cases, then a tag bit is added -to the enum's representation. The tag bit is set for the no-data cases, which -are then assigned values in the data area of the enum in declaration order. - -:: - - enum IntOrInfinity { => LLVM <{ i64, i1 }> - case NegInfinity => <{ i64, i1 }> { 0, 1 } - case Int(Int) => <{ i64, i1 }> { %Int, 0 } - case PosInfinity => <{ i64, i1 }> { 1, 1 } - } - - IntOrInfinity.Int( 0) => <{ i64, i1 }> { 0, 0 } - IntOrInfinity.Int(20721) => <{ i64, i1 }> { 20721, 0 } - -Multi-Payload Enums -``````````````````` - -If an enum has more than one case with data type, then a tag is necessary to -discriminate the data types. The ABI will first try to find common -**spare bits**, that is, bits in the data types' binary representations which are -either fixed-zero or ignored by valid values of all of the data types. The tag -will be scattered into these spare bits as much as possible. Currently only -spare bits of primitive integer types, such as the high bits of an ``i21`` -type, are considered. The enum data is represented as an integer with the -storage size in bits of the largest data type. - -:: - - enum TerminalChar { => LLVM i32 - case Plain(UnicodeScalar) => i32 (zext i21 %Plain to i32) - case Bold(UnicodeScalar) => i32 (or (zext i21 %Bold to i32), 0x0020_0000) - case Underline(UnicodeScalar) => i32 (or (zext i21 %Underline to i32), 0x0040_0000) - case Blink(UnicodeScalar) => i32 (or (zext i21 %Blink to i32), 0x0060_0000) - case Empty => i32 0x0080_0000 - case Cursor => i32 0x0080_0001 - } - -If there are not enough spare bits to contain the tag, then additional bits are -added to the representation to contain the tag. Tag values are -assigned to data cases in declaration order. If there are no-data cases, they -are collected under a common tag, and assigned values in the data area of the -enum in declaration order. - -:: - - class Bignum {} - - enum IntDoubleOrBignum { => LLVM <{ i64, i2 }> - case Int(Int) => <{ i64, i2 }> { %Int, 0 } - case Double(Double) => <{ i64, i2 }> { (bitcast %Double to i64), 1 } - case Bignum(Bignum) => <{ i64, i2 }> { (ptrtoint %Bignum to i64), 2 } - } - -Existential Container Layout -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Values of protocol type, protocol composition type, or "any" type -(``protocol<>``) are laid out using **existential containers** (so-called -because these types are "existential types" in type theory). - -Opaque Existential Containers -````````````````````````````` - -If there is no class constraint on a protocol or protocol composition type, -the existential container has to accommodate a value of arbitrary size and -alignment. It does this using a **fixed-size buffer**, which is three pointers -in size and pointer-aligned. This either directly contains the value, if its -size and alignment are both less than or equal to the fixed-size buffer's, or -contains a pointer to a side allocation owned by the existential container. -The type of the contained value is identified by its `type metadata` record, -and witness tables for all of the required protocol conformances are included. -The layout is as if declared in the following C struct:: - - struct OpaqueExistentialContainer { - void *fixedSizeBuffer[3]; - Metadata *type; - WitnessTable *witnessTables[NUM_WITNESS_TABLES]; - }; - -Class Existential Containers -```````````````````````````` - -If one or more of the protocols in a protocol or protocol composition type -have a class constraint, then only class values can be stored in the existential -container, and a more efficient representation is used. Class instances are -always a single pointer in size, so a fixed-size buffer and potential side -allocation is not needed, and class instances always have a reference to their -own type metadata, so the separate metadata record is not needed. The -layout is thus as if declared in the following C struct:: - - struct ClassExistentialContainer { - HeapObject *value; - WitnessTable *witnessTables[NUM_WITNESS_TABLES]; - }; - -Note that if no witness tables are needed, such as for the "any class" type -``protocol`` or an Objective-C protocol type, then the only element of -the layout is the heap object pointer. This is ABI-compatible with ``id`` -and ``id `` types in Objective-C. - -Type Metadata -------------- - -The Swift runtime keeps a **metadata record** for every type used in a program, -including every instantiation of generic types. These metadata records can -be used by (TODO: reflection and) debugger tools to discover information about -types. For non-generic nominal types, these metadata records are generated -statically by the compiler. For instances of generic types, and for intrinsic -types such as tuples, functions, protocol compositions, etc., metadata records -are lazily created by the runtime as required. Every type has a unique metadata -record; two **metadata pointer** values are equal iff the types are equivalent. - -In the layout descriptions below, offsets are given relative to the -metadata pointer as an index into an array of pointers. On a 32-bit platform, -**offset 1** means an offset of 4 bytes, and on 64-bit platforms, it means -an offset of 8 bytes. - -Common Metadata Layout -~~~~~~~~~~~~~~~~~~~~~~ - -All metadata records share a common header, with the following fields: - -- The **value witness table** pointer references a vtable of functions - that implement the value semantics of the type, providing fundamental - operations such as allocating, copying, and destroying values of the type. - The value witness table also records the size, alignment, stride, and other - fundamental properties of the type. The value witness table pointer is at - **offset -1** from the metadata pointer, that is, the pointer-sized word - **immediately before** the pointer's referenced address. - -- The **kind** field is a pointer-sized integer that describes the kind of type - the metadata describes. This field is at **offset 0** from the metadata - pointer. - - The current kind values are as follows: - - * `Struct metadata`_ has a kind of **1**. - * `Enum metadata`_ has a kind of **2**. - * **Opaque metadata** has a kind of **8**. This is used for compiler - ``Builtin`` primitives that have no additional runtime information. - * `Tuple metadata`_ has a kind of **9**. - * `Function metadata`_ has a kind of **10**. - * `Protocol metadata`_ has a kind of **12**. This is used for - protocol types, for protocol compositions, and for the "any" type - ``protocol<>``. - * `Metatype metadata`_ has a kind of **13**. - * `Class metadata`_, instead of a kind, has an *isa pointer* in its kind slot, - pointing to the class's metaclass record. This isa pointer is guaranteed - to have an integer value larger than **4096** and so can be discriminated - from non-class kind values. - -Struct Metadata -~~~~~~~~~~~~~~~ - -In addition to the `common metadata layout`_ fields, struct metadata records -contain the following fields: - -- The `nominal type descriptor`_ is referenced at **offset 1**. - -- A reference to the **parent** metadata record is stored at **offset 2**. For - structs that are members of an enclosing nominal type, this is a reference - to the enclosing type's metadata. For top-level structs, this is null. - - TODO: The parent pointer is currently always null. - -- A vector of **field offsets** begins at **offset 3**. For each field of the - struct, in ``var`` declaration order, the field's offset in bytes from the - beginning of the struct is stored as a pointer-sized integer. - -- If the struct is generic, then the - `generic parameter vector`_ begins at **offset 3+n**, where **n** is the - number of fields in the struct. - -Enum Metadata -~~~~~~~~~~~~~ - -In addition to the `common metadata layout`_ fields, enum metadata records -contain the following fields: - -- The `nominal type descriptor`_ is referenced at **offset 1**. - -- A reference to the **parent** metadata record is stored at **offset 2**. For - enums that are members of an enclosing nominal type, this is a reference to - the enclosing type's metadata. For top-level enums, this is null. - - TODO: The parent pointer is currently always null. - -- If the enum is generic, then the - `generic parameter vector`_ begins at **offset 3**. - -Tuple Metadata -~~~~~~~~~~~~~~ - -In addition to the `common metadata layout`_ fields, tuple metadata records -contain the following fields: - -- The **number of elements** in the tuple is a pointer-sized integer at - **offset 1**. -- The **labels string** is a pointer to a list of consecutive null-terminated - label names for the tuple at **offset 2**. Each label name is given as a - null-terminated, UTF-8-encoded string in sequence. If the tuple has no - labels, this is a null pointer. - - TODO: The labels string pointer is currently always null, and labels are - not factored into tuple metadata uniquing. - -- The **element vector** begins at **offset 3** and consists of a vector of - type–offset pairs. The metadata for the *n*\ th element's type is a pointer - at **offset 3+2*n**. The offset in bytes from the beginning of the tuple to - the beginning of the *n*\ th element is at **offset 3+2*n+1**. - -Function Metadata -~~~~~~~~~~~~~~~~~ - -In addition to the `common metadata layout`_ fields, function metadata records -contain the following fields: - -- The number of arguments to the function is stored at **offset 1**. -- A reference to the **result type** metadata record is stored at - **offset 2**. If the function has multiple returns, this references a - `tuple metadata`_ record. -- The **argument vector** begins at **offset 3** and consists of pointers to - metadata records of the function's arguments. - - If the function takes any **inout** arguments, a pointer to each argument's - metadata record will be appended separately, the lowest bit being set if it is - **inout**. Because of pointer alignment, the lowest bit will always be free to - hold this tag. - - If the function takes no **inout** arguments, there will be only one pointer in - the vector for the following cases: - - * 0 arguments: a `tuple metadata`_ record for the empty tuple - * 1 argument: the first and only argument's metadata record - * >1 argument: a `tuple metadata`_ record containing the arguments - -Protocol Metadata -~~~~~~~~~~~~~~~~~ - -In addition to the `common metadata layout`_ fields, protocol metadata records -contain the following fields: - -- A **layout flags** word is stored at **offset 1**. The bits of this word - describe the `existential container layout`_ used to represent - values of the type. The word is laid out as follows: - - * The **number of witness tables** is stored in the least significant 31 bits. - Values of the protocol type contain this number of witness table pointers - in their layout. - * The **class constraint** is stored at bit 31. This bit is set if the type - is **not** class-constrained, meaning that struct, enum, or class values - can be stored in the type. If not set, then only class values can be stored - in the type, and the type uses a more efficient layout. - - Note that the field is pointer-sized, even though only the lowest 32 bits are - currently inhabited on all platforms. These values can be derived from the - `protocol descriptor`_ records, but are pre-calculated for convenience. - -- The **number of protocols** that make up the protocol composition is stored at - **offset 2**. For the "any" types ``protocol<>`` or ``protocol``, this - is zero. For a single-protocol type ``P``, this is one. For a protocol - composition type ``protocol``, this is the number of protocols. - -- The **protocol descriptor vector** begins at **offset 3**. This is an inline - array of pointers to the `protocol descriptor`_ for every protocol in the - composition, or the single protocol descriptor for a protocol type. For - an "any" type, there is no protocol descriptor vector. - -Metatype Metadata -~~~~~~~~~~~~~~~~~ - -In addition to the `common metadata layout`_ fields, metatype metadata records -contain the following fields: - -- A reference to the metadata record for the **instance type** that the metatype - represents is stored at **offset 1**. - -Class Metadata -~~~~~~~~~~~~~~ - -Class metadata is designed to interoperate with Objective-C; all class metadata -records are also valid Objective-C ``Class`` objects. Class metadata pointers -are used as the values of class metatypes, so a derived class's metadata -record also serves as a valid class metatype value for all of its ancestor -classes. - -- The **destructor pointer** is stored at **offset -2** from the metadata - pointer, behind the value witness table. This function is invoked by Swift's - deallocator when the class instance is destroyed. -- The **isa pointer** pointing to the class's Objective-C-compatible metaclass - record is stored at **offset 0**, in place of an integer kind discriminator. -- The **super pointer** pointing to the metadata record for the superclass is - stored at **offset 1**. If the class is a root class, it is null. -- Two words are reserved for use by the Objective-C runtime at **offset 2** - and **offset 3**. -- The **rodata pointer** is stored at **offset 4**; it points to an Objective-C - compatible rodata record for the class. This pointer value includes a tag. - The **low bit is always set to 1** for Swift classes and always set to 0 for - Objective-C classes. -- The **class flags** are a 32-bit field at **offset 5**. -- The **instance address point** is a 32-bit field following the class flags. - A pointer to an instance of this class points this number of bytes after the - beginning of the instance. -- The **instance size** is a 32-bit field following the instance address point. - This is the number of bytes of storage present in every object of this type. -- The **instance alignment mask** is a 16-bit field following the instance size. - This is a set of low bits which must not be set in a pointer to an instance - of this class. -- The **runtime-reserved field** is a 16-bit field following the instance - alignment mask. The compiler initializes this to zero. -- The **class object size** is a 32-bit field following the runtime-reserved - field. This is the total number of bytes of storage in the class metadata - object. -- The **class object address point** is a 32-bit field following the class - object size. This is the number of bytes of storage in the class metadata - object. -- The `nominal type descriptor`_ for the most-derived class type is referenced - at an offset immediately following the class object address point. This is - **offset 8** on a 64-bit platform or **offset 11** on a 32-bit platform. -- For each Swift class in the class's inheritance hierarchy, in order starting - from the root class and working down to the most derived class, the following - fields are present: - - * First, a reference to the **parent** metadata record is stored. - For classes that are members of an enclosing nominal type, this is a - reference to the enclosing type's metadata. For top-level classes, this is - null. - - TODO: The parent pointer is currently always null. - - * If the class is generic, its `generic parameter vector`_ is stored inline. - * The **vtable** is stored inline and contains a function pointer to the - implementation of every method of the class in declaration order. - * If the layout of a class instance is dependent on its generic parameters, - then a **field offset vector** is stored inline, containing offsets in - bytes from an instance pointer to each field of the class in declaration - order. (For classes with fixed layout, the field offsets are accessible - statically from global variables, similar to Objective-C ivar offsets.) - - Note that none of these fields are present for Objective-C base classes in - the inheritance hierarchy. - -Generic Parameter Vector -~~~~~~~~~~~~~~~~~~~~~~~~ - -Metadata records for instances of generic types contain information about their -generic parameters. For each parameter of the type, a reference to the metadata -record for the type argument is stored. After all of the type argument -metadata references, for each type parameter, if there are protocol -requirements on that type parameter, a reference to the witness table for each -protocol it is required to conform to is stored in declaration order. - -For example, given a generic type with the parameters ````, its -generic parameter record will consist of references to the metadata records -for ``T``, ``U``, and ``V`` in succession, as if laid out in a C struct:: - - struct GenericParameterVector { - TypeMetadata *T, *U, *V; - }; - -If we add protocol requirements to the parameters, for example, -``, V>``, then the type's generic -parameter vector contains witness tables for those protocols, as if laid out:: - - struct GenericParameterVector { - TypeMetadata *T, *U, *V; - RuncibleWitnessTable *T_Runcible; - FungibleWitnessTable *U_Fungible; - AnsibleWitnessTable *U_Ansible; - }; - -Nominal Type Descriptor -~~~~~~~~~~~~~~~~~~~~~~~ - -The metadata records for class, struct, and enum types contain a pointer to a -**nominal type descriptor**, which contains basic information about the nominal -type such as its name, members, and metadata layout. For a generic type, one -nominal type descriptor is shared for all instantiations of the type. The -layout is as follows: - -- The **kind** of type is stored at **offset 0**, which is as follows: - - * **0** for a class, - * **1** for a struct, or - * **2** for an enum. - -- The mangled **name** is referenced as a null-terminated C string at - **offset 1**. This name includes no bound generic parameters. -- The following four fields depend on the kind of nominal type. - - * For a struct or class: - - + The **number of fields** is stored at **offset 2**. This is the length - of the field offset vector in the metadata record, if any. - + The **offset to the field offset vector** is stored at **offset 3**. - This is the offset in pointer-sized words of the field offset vector for - the type in the metadata record. If no field offset vector is stored - in the metadata record, this is zero. - + The **field names** are referenced as a doubly-null-terminated list of - C strings at **offset 4**. The order of names corresponds to the order - of fields in the field offset vector. - + The **field type accessor** is a function pointer at **offset 5**. If - non-null, the function takes a pointer to an instance of type metadata - for the nominal type, and returns a pointer to an array of type metadata - references for the types of the fields of that instance. The order matches - that of the field offset vector and field name list. - - * For an enum: - - + The **number of payload cases** and **payload size offset** are stored - at **offset 2**. The least significant 24 bits are the number of payload - cases, and the most significant 8 bits are the offset of the payload - size in the type metadata, if present. - + The **number of no-payload cases** is stored at **offset 3**. - + The **case names** are referenced as a doubly-null-terminated list of - C strings at **offset 4**. The names are ordered such that payload cases - come first, followed by no-payload cases. Within each half of the list, - the order of names corresponds to the order of cases in the enum - declaration. - + The **case type accessor** is a function pointer at **offset 5**. If - non-null, the function takes a pointer to an instance of type metadata - for the enum, and returns a pointer to an array of type metadata - references for the types of the cases of that instance. The order matches - that of the case name list. This function is similar to the field type - accessor for a struct, except also the least significant bit of each - element in the result is set if the enum case is an **indirect case**. - -- If the nominal type is generic, a pointer to the **metadata pattern** that - is used to form instances of the type is stored at **offset 6**. The pointer - is null if the type is not generic. - -- The **generic parameter descriptor** begins at **offset 7**. This describes - the layout of the generic parameter vector in the metadata record: - - * The **offset of the generic parameter vector** is stored at **offset 7**. - This is the offset in pointer-sized words of the generic parameter vector - inside the metadata record. If the type is not generic, this is zero. - * The **number of type parameters** is stored at **offset 8**. This count - includes associated types of type parameters with protocol constraints. - * The **number of type parameters** is stored at **offset 9**. This count - includes only the primary formal type parameters. - * For each type parameter **n**, the following fields are stored: - - + The **number of witnesses** for the type parameter is stored at - **offset 10+n**. This is the number of witness table pointers that are - stored for the type parameter in the generic parameter vector. - -Note that there is no nominal type descriptor for protocols or protocol types. -See the `protocol descriptor`_ description below. - -Protocol Descriptor -~~~~~~~~~~~~~~~~~~~ - -`Protocol metadata` contains references to zero, one, or more **protocol -descriptors** that describe the protocols values of the type are required to -conform to. The protocol descriptor is laid out to be compatible with -Objective-C ``Protocol`` objects. The layout is as follows: - -- An **isa** placeholder is stored at **offset 0**. This field is populated by - the Objective-C runtime. -- The mangled **name** is referenced as a null-terminated C string at - **offset 1**. -- If the protocol inherits one or more other protocols, a pointer to the - **inherited protocols list** is stored at **offset 2**. The list starts with - the number of inherited protocols as a pointer-sized integer, and is followed - by that many protocol descriptor pointers. If the protocol inherits no other - protocols, this pointer is null. -- For an ObjC-compatible protocol, its **required instance methods** are stored - at **offset 3** as an ObjC-compatible method list. This is null for native - Swift protocols. -- For an ObjC-compatible protocol, its **required class methods** are stored - at **offset 4** as an ObjC-compatible method list. This is null for native - Swift protocols. -- For an ObjC-compatible protocol, its **optional instance methods** are stored - at **offset 5** as an ObjC-compatible method list. This is null for native - Swift protocols. -- For an ObjC-compatible protocol, its **optional class methods** are stored - at **offset 6** as an ObjC-compatible method list. This is null for native - Swift protocols. -- For an ObjC-compatible protocol, its **instance properties** are stored - at **offset 7** as an ObjC-compatible property list. This is null for native - Swift protocols. -- The **size** of the protocol descriptor record is stored as a 32-bit integer - at **offset 8**. This is currently 72 on 64-bit platforms and 40 on 32-bit - platforms. -- **Flags** are stored as a 32-bit integer after the size. The following bits - are currently used (counting from least significant bit zero): - - * **Bit 0** is the **Swift bit**. It is set for all protocols defined in - Swift and unset for protocols defined in Objective-C. - * **Bit 1** is the **class constraint bit**. It is set if the protocol is - **not** class-constrained, meaning that any struct, enum, or class type - may conform to the protocol. It is unset if only classes can conform to - the protocol. (The inverted meaning is for compatibility with Objective-C - protocol records, in which the bit is never set. Objective-C protocols can - only be conformed to by classes.) - * **Bit 2** is the **witness table bit**. It is set if dispatch to the - protocol's methods is done through a witness table, which is either passed - as an extra parameter to generic functions or included in the `existential - container layout`_ of protocol types. It is unset if dispatch is done - through ``objc_msgSend`` and requires no additional information to accompany - a value of conforming type. - * **Bit 31** is set by the Objective-C runtime when it has done its - initialization of the protocol record. It is unused by the Swift runtime. - -Heap Objects ------------- - -Heap Metadata -~~~~~~~~~~~~~ - -Heap Object Header -~~~~~~~~~~~~~~~~~~ - -Mangling --------- -:: - - mangled-name ::= '_T' global - -All Swift-mangled names begin with this prefix. - -Globals -~~~~~~~ - -:: - - global ::= 't' type // standalone type (for DWARF) - global ::= 'M' type // type metadata (address point) - // -- type starts with [BCOSTV] - global ::= 'Mf' type // 'full' type metadata (start of object) - global ::= 'MP' type // type metadata pattern - global ::= 'Ma' type // type metadata access function - global ::= 'ML' type // type metadata lazy cache variable - global ::= 'Mm' type // class metaclass - global ::= 'Mn' nominal-type // nominal type descriptor - global ::= 'Mp' protocol // protocol descriptor - global ::= 'PA' .* // partial application forwarder - global ::= 'PAo' .* // ObjC partial application forwarder - global ::= 'w' value-witness-kind type // value witness - global ::= 'WV' type // value witness table - global ::= 'Wo' entity // witness table offset - global ::= 'Wv' directness entity // field offset - global ::= 'WP' protocol-conformance // protocol witness table - global ::= 'Wa' protocol-conformance // protocol witness table accessor - global ::= 'Wl' type protocol-conformance // lazy protocol witness table accessor - global ::= 'WL' protocol-conformance // lazy protocol witness table cache variable - global ::= 'WD' protocol-conformance // dependent proto witness table generator - global ::= 'Wd' protocol-conformance // dependent proto witness table template - global ::= entity // some identifiable thing - global ::= 'TO' global // ObjC-as-swift thunk - global ::= 'To' global // swift-as-ObjC thunk - global ::= 'TD' global // dynamic dispatch thunk - global ::= 'Td' global // direct method reference thunk - global ::= 'TR' reabstract-signature // reabstraction thunk helper function - global ::= 'Tr' reabstract-signature // reabstraction thunk - - global ::= 'TS' specializationinfo '_' mangled-name - specializationinfo ::= 'g' passid (type protocol-conformance* '_')+ // Generic specialization info. - specializationinfo ::= 'f' passid (funcspecializationarginfo '_')+ // Function signature specialization kind - passid ::= integer // The id of the pass that generated this specialization. - funcsigspecializationarginfo ::= 'cl' closurename type* // Closure specialized with closed over types in argument order. - funcsigspecializationarginfo ::= 'n' // Unmodified argument - funcsigspecializationarginfo ::= 'cp' funcsigspecializationconstantproppayload // Constant propagated argument - funcsigspecializationarginfo ::= 'd' // Dead argument - funcsigspecializationarginfo ::= 'g' 's'? // Owned => Guaranteed and Exploded if 's' present. - funcsigspecializationarginfo ::= 's' // Exploded - funcsigspecializationconstantpropinfo ::= 'fr' mangled-name - funcsigspecializationconstantpropinfo ::= 'g' mangled-name - funcsigspecializationconstantpropinfo ::= 'i' 64-bit-integer - funcsigspecializationconstantpropinfo ::= 'fl' float-as-64-bit-integer - funcsigspecializationconstantpropinfo ::= 'se' stringencoding 'v' md5hash - - global ::= 'TV' global // vtable override thunk - global ::= 'TW' protocol-conformance entity - // protocol witness thunk - entity ::= nominal-type // named type declaration - entity ::= static? entity-kind context entity-name - entity-kind ::= 'F' // function (ctor, accessor, etc.) - entity-kind ::= 'v' // variable (let/var) - entity-kind ::= 'i' // subscript ('i'ndex) itself (not the individual accessors) - entity-kind ::= 'I' // initializer - entity-name ::= decl-name type // named declaration - entity-name ::= 'A' index // default argument generator - entity-name ::= 'a' addressor-kind decl-name type // mutable addressor - entity-name ::= 'C' type // allocating constructor - entity-name ::= 'c' type // non-allocating constructor - entity-name ::= 'D' // deallocating destructor; untyped - entity-name ::= 'd' // non-deallocating destructor; untyped - entity-name ::= 'g' decl-name type // getter - entity-name ::= 'i' // non-local variable initializer - entity-name ::= 'l' addressor-kind decl-name type // non-mutable addressor - entity-name ::= 'm' decl-name type // materializeForSet - entity-name ::= 's' decl-name type // setter - entity-name ::= 'U' index type // explicit anonymous closure expression - entity-name ::= 'u' index type // implicit anonymous closure - entity-name ::= 'w' decl-name type // willSet - entity-name ::= 'W' decl-name type // didSet - static ::= 'Z' // entity is a static member of a type - decl-name ::= identifier - decl-name ::= local-decl-name - decl-name ::= private-decl-name - local-decl-name ::= 'L' index identifier // locally-discriminated declaration - private-decl-name ::= 'P' identifier identifier // file-discriminated declaration - reabstract-signature ::= ('G' generic-signature)? type type - addressor-kind ::= 'u' // unsafe addressor (no owner) - addressor-kind ::= 'O' // owning addressor (non-native owner) - addressor-kind ::= 'o' // owning addressor (native owner) - addressor-kind ::= 'p' // pinning addressor (native owner) - -An ``entity`` starts with a ``nominal-type-kind`` (``[COPV]``), a -substitution (``[Ss]``) of a nominal type, or an ``entity-kind`` -(``[FIiv]``). - -An ``entity-name`` starts with ``[AaCcDggis]`` or a ``decl-name``. -A ``decl-name`` starts with ``[LP]`` or an ``identifier`` (``[0-9oX]``). - -A ``context`` starts with either an ``entity``, an ``extension`` (which starts -with ``[Ee]``), or a ``module``, which might be an ``identifier`` (``[0-9oX]``) -or a substitution of a module (``[Ss]``). - -A global mangling starts with an ``entity`` or ``[MTWw]``. - -If a partial application forwarder is for a static symbol, its name will -start with the sequence ``_TPA_`` followed by the mangled symbol name of the -forwarder's destination. - -A generic specialization mangling consists of a header, specifying the types -and conformances used to specialize the generic function, followed by the -full mangled name of the original unspecialized generic symbol. - -The first identifier in a ```` is a string that represents -the file the original declaration came from. It should be considered unique -within the enclosing module. The second identifier is the name of the entity. - -Not all declarations marked ``private`` declarations will use the -```` mangling; if the entity's context is enough to uniquely -identify the entity, the simple ``identifier`` form is preferred. - -The types in a ```` are always non-polymorphic -```` types. - -Direct and Indirect Symbols -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -:: - - directness ::= 'd' // direct - directness ::= 'i' // indirect - -A direct symbol resolves directly to the address of an object. An -indirect symbol resolves to the address of a pointer to the object. -They are distinct manglings to make a certain class of bugs -immediately obvious. - -The terminology is slightly overloaded when discussing offsets. A -direct offset resolves to a variable holding the true offset. An -indirect offset resolves to a variable holding an offset to be applied -to type metadata to get the address of the true offset. (Offset -variables are required when the object being accessed lies within a -resilient structure. When the layout of the object may depend on -generic arguments, these offsets must be kept in metadata. Indirect -field offsets are therefore required when accessing fields in generic -types where the metadata itself has unknown layout.) - -Declaration Contexts -~~~~~~~~~~~~~~~~~~~~ - -:: - - context ::= module - context ::= extension - context ::= entity - module ::= substitution // other substitution - module ::= identifier // module name - module ::= known-module // abbreviation - extension ::= 'E' module entity - extension ::= 'e' module generic-signature entity - -These manglings identify the enclosing context in which an entity was declared, -such as its enclosing module, function, or nominal type. - -An ``extension`` mangling is used whenever an entity's declaration context is -an extension *and* the entity being extended is in a different module. In this -case the extension's module is mangled first, followed by the entity being -extended. If the extension and the extended entity are in the same module, the -plain ``entity`` mangling is preferred. If the extension is constrained, the -constraints on the extension are mangled in its generic signature. - -When mangling the context of a local entity within a constructor or -destructor, the non-allocating or non-deallocating variant is used. - -Types -~~~~~ - -:: - - type ::= 'Bb' // Builtin.BridgeObject - type ::= 'BB' // Builtin.UnsafeValueBuffer - type ::= 'Bf' natural '_' // Builtin.Float - type ::= 'Bi' natural '_' // Builtin.Int - type ::= 'BO' // Builtin.ObjCPointer - type ::= 'Bo' // Builtin.ObjectPointer - type ::= 'Bp' // Builtin.RawPointer - type ::= 'Bv' natural type // Builtin.Vecx - type ::= 'Bw' // Builtin.Word - type ::= nominal-type - type ::= associated-type - type ::= 'a' context identifier // Type alias (DWARF only) - type ::= 'b' type type // objc block function type - type ::= 'c' type type // C function pointer type - type ::= 'F' throws-annotation? type type // function type - type ::= 'f' throws-annotation? type type // uncurried function type - type ::= 'G' type + '_' // generic type application - type ::= 'K' type type // @auto_closure function type - type ::= 'M' type // metatype without representation - type ::= 'XM' metatype-repr type // metatype with representation - type ::= 'P' protocol-list '_' // protocol type - type ::= 'PM' type // existential metatype without representation - type ::= 'XPM' metatype-repr type // existential metatype with representation - type ::= archetype - type ::= 'R' type // inout - type ::= 'T' tuple-element* '_' // tuple - type ::= 't' tuple-element* '_' // variadic tuple - type ::= 'Xo' type // @unowned type - type ::= 'Xu' type // @unowned(unsafe) type - type ::= 'Xw' type // @weak type - type ::= 'XF' impl-function-type // function implementation type - type ::= 'Xf' type type // @thin function type - nominal-type ::= known-nominal-type - nominal-type ::= substitution - nominal-type ::= nominal-type-kind declaration-name - nominal-type-kind ::= 'C' // class - nominal-type-kind ::= 'O' // enum - nominal-type-kind ::= 'V' // struct - archetype ::= 'Q' index // archetype with depth=0, idx=N - archetype ::= 'Qd' index index // archetype with depth=M+1, idx=N - archetype ::= associated-type - archetype ::= qualified-archetype - associated-type ::= substitution - associated-type ::= 'Q' protocol-context // self type of protocol - associated-type ::= 'Q' archetype identifier // associated type - qualified-archetype ::= 'Qq' index context // archetype+context (DWARF only) - protocol-context ::= 'P' protocol - tuple-element ::= identifier? type - metatype-repr ::= 't' // Thin metatype representation - metatype-repr ::= 'T' // Thick metatype representation - metatype-repr ::= 'o' // ObjC metatype representation - throws-annotation ::= 'z' // 'throws' annotation on function types - - - type ::= 'u' generic-signature type // generic type - type ::= 'x' // generic param, depth=0, idx=0 - type ::= 'q' generic-param-index // dependent generic parameter - type ::= 'q' type assoc-type-name // associated type of non-generic param - type ::= 'w' generic-param-index assoc-type-name // associated type - type ::= 'W' generic-param-index assoc-type-name+ '_' // associated type at depth - - generic-param-index ::= 'x' // depth = 0, idx = 0 - generic-param-index ::= index // depth = 0, idx = N+1 - generic-param-index ::= 'd' index index // depth = M+1, idx = N - -```` never begins or ends with a number. -```` never begins with an underscore. -```` never begins with ``d``. -```` never begins with ``z``. - -Note that protocols mangle differently as types and as contexts. A protocol -context always consists of a single protocol name and so mangles without a -trailing underscore. A protocol type can have zero, one, or many protocol bounds -which are juxtaposed and terminated with a trailing underscore. - -:: - - assoc-type-name ::= ('P' protocol-name)? identifier - assoc-type-name ::= substitution - -Associated types use an abbreviated mangling when the base generic parameter -or associated type is constrained by a single protocol requirement. The -associated type in this case can be referenced unambiguously by name alone. -If the base has multiple conformance constraints, then the protocol name is -mangled in to disambiguate. - -:: - - impl-function-type ::= - impl-callee-convention impl-function-attribute* generic-signature? '_' - impl-parameter* '_' impl-result* '_' - impl-callee-convention ::= 't' // thin - impl-callee-convention ::= impl-convention // thick, callee transferred with given convention - impl-convention ::= 'a' // direct, autoreleased - impl-convention ::= 'd' // direct, no ownership transfer - impl-convention ::= 'D' // direct, no ownership transfer, - // dependent on 'self' parameter - impl-convention ::= 'g' // direct, guaranteed - impl-convention ::= 'e' // direct, deallocating - impl-convention ::= 'i' // indirect, ownership transfer - impl-convention ::= 'l' // indirect, inout - impl-convention ::= 'G' // indirect, guaranteed - impl-convention ::= 'o' // direct, ownership transfer - impl-convention ::= 'z' impl-convention // error result - impl-function-attribute ::= 'Cb' // compatible with C block invocation function - impl-function-attribute ::= 'Cc' // compatible with C global function - impl-function-attribute ::= 'Cm' // compatible with Swift method - impl-function-attribute ::= 'CO' // compatible with ObjC method - impl-function-attribute ::= 'Cw' // compatible with protocol witness - impl-function-attribute ::= 'N' // noreturn - impl-function-attribute ::= 'G' // generic - impl-parameter ::= impl-convention type - impl-result ::= impl-convention type - -For the most part, manglings follow the structure of formal language -types. However, in some cases it is more useful to encode the exact -implementation details of a function type. - -Any ```` productions must appear in the order -in which they are specified above: e.g. a noreturn C function is -mangled with ``CcN``. - -Note that the convention and function-attribute productions do not -need to be disambiguated from the start of a ````. - -Generics -~~~~~~~~ - -:: - - protocol-conformance ::= ('u' generic-signature)? type protocol module - -```` refers to a type's conformance to a protocol. The -named module is the one containing the extension or type declaration that -declared the conformance. - -:: - - generic-signature ::= (generic-param-count+)? ('R' requirement*)? 'r' - generic-param-count ::= 'z' // zero parameters - generic-param-count ::= index // N+1 parameters - requirement ::= type-param protocol-name // protocol requirement - requirement ::= type-param type // base class requirement - // type starts with [CS] - requirement ::= type-param 'z' type // 'z'ame-type requirement - - // Special type mangling for type params that saves the initial 'q' on - // generic params - type-param ::= generic-param-index // generic parameter - type-param ::= 'w' generic-param-index assoc-type-name // associated type - type-param ::= 'W' generic-param-index assoc-type-name+ '_' - -A generic signature begins by describing the number of generic parameters at -each depth of the signature, followed by the requirements. As a special case, -no ``generic-param-count`` values indicates a single generic parameter at -the outermost depth:: - - urFq_q_ // T_0_0 -> T_0_0 - u_0_rFq_qd_0_ // T_0_0 -> T_1_1 - -Value Witnesses -~~~~~~~~~~~~~~~ - -TODO: document these - -:: - - value-witness-kind ::= 'al' // allocateBuffer - value-witness-kind ::= 'ca' // assignWithCopy - value-witness-kind ::= 'ta' // assignWithTake - value-witness-kind ::= 'de' // deallocateBuffer - value-witness-kind ::= 'xx' // destroy - value-witness-kind ::= 'XX' // destroyBuffer - value-witness-kind ::= 'Xx' // destroyArray - value-witness-kind ::= 'CP' // initializeBufferWithCopyOfBuffer - value-witness-kind ::= 'Cp' // initializeBufferWithCopy - value-witness-kind ::= 'cp' // initializeWithCopy - value-witness-kind ::= 'TK' // initializeBufferWithTakeOfBuffer - value-witness-kind ::= 'Tk' // initializeBufferWithTake - value-witness-kind ::= 'tk' // initializeWithTake - value-witness-kind ::= 'pr' // projectBuffer - value-witness-kind ::= 'xs' // storeExtraInhabitant - value-witness-kind ::= 'xg' // getExtraInhabitantIndex - value-witness-kind ::= 'Cc' // initializeArrayWithCopy - value-witness-kind ::= 'Tt' // initializeArrayWithTakeFrontToBack - value-witness-kind ::= 'tT' // initializeArrayWithTakeBackToFront - value-witness-kind ::= 'ug' // getEnumTag - value-witness-kind ::= 'up' // destructiveProjectEnumData - -```` differentiates the kinds of value -witness functions for a type. - -Identifiers -~~~~~~~~~~~ - -:: - - identifier ::= natural identifier-start-char identifier-char* - identifier ::= 'o' operator-fixity natural operator-char+ - - operator-fixity ::= 'p' // prefix operator - operator-fixity ::= 'P' // postfix operator - operator-fixity ::= 'i' // infix operator - - operator-char ::= 'a' // & 'and' - operator-char ::= 'c' // @ 'commercial at' - operator-char ::= 'd' // / 'divide' - operator-char ::= 'e' // = 'equals' - operator-char ::= 'g' // > 'greater' - operator-char ::= 'l' // < 'less' - operator-char ::= 'm' // * 'multiply' - operator-char ::= 'n' // ! 'not' - operator-char ::= 'o' // | 'or' - operator-char ::= 'p' // + 'plus' - operator-char ::= 'q' // ? 'question' - operator-char ::= 'r' // % 'remainder' - operator-char ::= 's' // - 'subtract' - operator-char ::= 't' // ~ 'tilde' - operator-char ::= 'x' // ^ 'xor' - operator-char ::= 'z' // . 'zperiod' - -```` is run-length encoded: the natural indicates how many -characters follow. Operator characters are mapped to letter characters as -given. In neither case can an identifier start with a digit, so -there's no ambiguity with the run-length. - -:: - - identifier ::= 'X' natural identifier-start-char identifier-char* - identifier ::= 'X' 'o' operator-fixity natural identifier-char* - -Identifiers that contain non-ASCII characters are encoded using the Punycode -algorithm specified in RFC 3492, with the modifications that ``_`` is used -as the encoding delimiter, and uppercase letters A through J are used in place -of digits 0 through 9 in the encoding character set. The mangling then -consists of an ``X`` followed by the run length of the encoded string and the -encoded string itself. For example, the identifier ``vergüenza`` is mangled -to ``X12vergenza_JFa``. (The encoding in standard Punycode would be -``vergenza-95a``) - -Operators that contain non-ASCII characters are mangled by first mapping the -ASCII operator characters to letters as for pure ASCII operator names, then -Punycode-encoding the substituted string. The mangling then consists of -``Xo`` followed by the fixity, run length of the encoded string, and the encoded -string itself. For example, the infix operator ``«+»`` is mangled to -``Xoi7p_qcaDc`` (``p_qcaDc`` being the encoding of the substituted -string ``«p»``). - -Substitutions -~~~~~~~~~~~~~ - -:: - - substitution ::= 'S' index - -```` is a back-reference to a previously mangled entity. The mangling -algorithm maintains a mapping of entities to substitution indices as it runs. -When an entity that can be represented by a substitution (a module, nominal -type, or protocol) is mangled, a substitution is first looked for in the -substitution map, and if it is present, the entity is mangled using the -associated substitution index. Otherwise, the entity is mangled normally, and -it is then added to the substitution map and associated with the next -available substitution index. - -For example, in mangling a function type -``(zim.zang.zung, zim.zang.zung, zim.zippity) -> zim.zang.zoo`` (with module -``zim`` and class ``zim.zang``), -the recurring contexts ``zim``, ``zim.zang``, and ``zim.zang.zung`` -will be mangled using substitutions after being mangled -for the first time. The first argument type will mangle in long form, -``CC3zim4zang4zung``, and in doing so, ``zim`` will acquire substitution ``S_``, -``zim.zang`` will acquire substitution ``S0_``, and ``zim.zang.zung`` will -acquire ``S1_``. The second argument is the same as the first and will mangle -using its substitution, ``CS1_``. The -third argument type will mangle using the substitution for ``zim``, -``CS_7zippity``. (It also acquires substitution ``S2_`` which would be used -if it mangled again.) The result type will mangle using the substitution for -``zim.zang``, ``CS0_zoo`` (and acquire substitution ``S3_``). The full -function type thus mangles as ``fTCC3zim4zang4zungCS1_CS_7zippity_CS0_zoo``. - -:: - - substitution ::= 's' - -The special substitution ``s`` is used for the ``Swift`` standard library -module. - -Predefined Substitutions -~~~~~~~~~~~~~~~~~~~~~~~~ - -:: - - known-module ::= 's' // Swift - known-module ::= 'SC' // C - known-module ::= 'So' // Objective-C - known-nominal-type ::= 'Sa' // Swift.Array - known-nominal-type ::= 'Sb' // Swift.Bool - known-nominal-type ::= 'Sc' // Swift.UnicodeScalar - known-nominal-type ::= 'Sd' // Swift.Float64 - known-nominal-type ::= 'Sf' // Swift.Float32 - known-nominal-type ::= 'Si' // Swift.Int - known-nominal-type ::= 'SP' // Swift.UnsafePointer - known-nominal-type ::= 'Sp' // Swift.UnsafeMutablePointer - known-nominal-type ::= 'SQ' // Swift.ImplicitlyUnwrappedOptional - known-nominal-type ::= 'Sq' // Swift.Optional - known-nominal-type ::= 'SR' // Swift.UnsafeBufferPointer - known-nominal-type ::= 'Sr' // Swift.UnsafeMutableBufferPointer - known-nominal-type ::= 'SS' // Swift.String - known-nominal-type ::= 'Su' // Swift.UInt - -```` and ```` are built-in substitutions for -certain common entities. Like any other substitution, they all start -with 'S'. - -The Objective-C module is used as the context for mangling Objective-C -classes as ````\ s. - -Indexes -~~~~~~~ - -:: - - index ::= '_' // 0 - index ::= natural '_' // N+1 - natural ::= [0-9]+ - -```` is a production for encoding numbers in contexts that can't -end in a digit; it's optimized for encoding smaller numbers. diff --git a/docs/ARCOptimization.md b/docs/ARCOptimization.md new file mode 100644 index 0000000000000..45c5b8d80f4c4 --- /dev/null +++ b/docs/ARCOptimization.md @@ -0,0 +1,216 @@ +ARC Optimization for Swift +========================== + +> **TODO** +> +> This is currently a place holder for design documentation on ARC +> optimization. + +Reference Counting Instructions +------------------------------- + +- strong\_retain +- strong\_retain\_autoreleased +- strong\_release +- strong\_retain\_unowned +- unowned\_retain +- unowned\_release +- load\_weak +- store\_weak +- fix\_lifetime +- mark\_dependence +- is\_unique +- is\_unique\_or\_pinned +- copy\_block + +Memory Behavior of ARC Operations +--------------------------------- + +At SIL level, reference counting and reference checking instructions are +attributed with MayHaveSideEffects to prevent arbitrary passes from +reordering them. + +At IR level, retains are marked NoModRef with respect to load and store +instructions so they don't pessimize memory dependence. (Note the +Retains are still considered to write to memory with respect to other +calls because getModRefBehavior is not overridden.) Releases cannot be +marked NoModRef because they can have arbitrary side effects. Is\_unique +calls cannot be marked NoModRef because they cannot be reordered with +other operations that may modify the reference count. + +> **TODO** +> +> Marking runtime calls with NoModRef in LLVM is misleading (they write +> memory), inconsistent (getModRefBehavior returns Unknown), and fragile +> (e.g. if we inline ARC operations at IR level). To be robust and allow +> stronger optimization, TBAA tags should be used to indicate functions +> that only access object metadata. This would also enable more LLVM +> level optimization in the presence of is\_unique checks which +> currently appear to arbitrarily write memory. + +RC Identity +----------- + +A core ARC concept in Swift optimization is the concept of +`Reference Count Identity` (RC Identity) and RC Identity preserving +instructions. An instruction `I` with n SSA arguments and m SSA results +is (i,j) RC Identity preserving if performing a `retain_value` on the +ith SSA argument immediately before `I` is executed is equivalent to +performing a `retain_value` on the jth SSA result of `I` immediately +following the execution of `I`. For example in the following, if: + + retain_value %x + %y = unary_instruction %x + +is equivalent to: + + %y = unary_instruction %x + retain_value %y + +then we say that unary\_instruction is a (0,0) RC Identity preserving +operation. In a case of a unary instruction, we omit (0,0) and just say +that the instruction is RC Identity preserving. + +In practice generally RC Identical operations are unary operations such +as casts. This would make it seem like RC Identity is an extension of +alias analysis. But RC Identity also has significantly more power than +alias analysis since: + +> - `struct` is an RC identity preserving operation if the `struct` +> literal only has one non-trivial operand. This means for instance +> that any struct with one reference counted field used as an owning +> pointer is RC Identical with its owning pointer (a useful property +> for Arrays). +> - An `enum` instruction is always RC Identical with the given +> tuple payload. +> - A `tuple` instruction is an RC identity preserving operation if +> the `tuple` literal has one non-trivial operand. +> - `init_class_existential` is an RC identity preserving operation +> since performing a retain\_value on a class existential is +> equivalent to performing a retain\_value on the class itself. + +The corresponding value projection operations have analogous properties. + +Given two SSA values `%a`, `%b`, we define `%a` as immediately RC +identical to `%b` if there exists an instruction `I` such that: + +- `%a` is the jth result of `I`. +- `%b` is the ith argument of `I`. +- `I` is (i,j) RC identity preserving. + +Easily the immediate RC identical relation must be reflexive and +symmetric but by its nature is not transitive. Then define the +equivalence relation RC Identity, `~rc`, by the relations that +`%a ~rc %b` if `%a` is immediately RC identical to `%b` or if there is a +finite sequence of n SSA values `{%a[i]}` such that `%a` is immediately +RC identical to `%a[0]` and `%b` is immediately RC identical to `%a[n]`. +We currently always assume that each equivalence class has one +dominating definition. + +These equivalence classes consisting of chains of RC identical values +are computed via the SILAnalysis called `RC Identity Analysis`. By +performing ARC optimization on RC Identical operations, our +optimizations are able to operate on the level of granularity that we +actually care about, ignoring superficial changes in SSA form that still +yield manipulations of the same reference count. + +*NOTE* RCIdentityAnalysis is a flow insensitive analysis. Dataflow that needs to + +: be flow sensitive must handle phi nodes in the dataflow itself. + +*NOTE* An important consequence of RC Identity is that value types with +only one RCIdentity are a simple case for ARC optimization to handle. +The ARC optimizer relies on other optimizations like SROA, Function +Signature Opts, and SimplifyCFG (for block arguments) to try and +eliminate cases where value types have multiple reference counted +subtypes. + +Copy-On-Write Considerations +---------------------------- + +The copy-on-write capabilities of some data structures, such as Array +and Set, are efficiently implemented via Builtin.isUnique calls which +lower directly to is\_unique instructions in SIL. + +The is\_unique instruction takes the address of a reference, and +although it does not actually change the reference, the reference must +appear mutable to the optimizer. This forces the optimizer to preserve a +retain distinct from what’s required to maintain lifetime for any of the +reference's source-level copies, because the called function is allowed +to replace the reference, thereby releasing the referent. Consider the +following sequence of rules: + +(1) An operation taking the address of a variable is allowed to replace + the reference held by that variable. The fact that is\_unique will + not actually replace it is opaque to the optimizer. +(2) If the refcount is 1 when the reference is replaced, the referent + is deallocated. +(3) A different source-level variable pointing at the same referent must + not be changed/invalidated by such a call. +(4) If such a variable exists, the compiler must guarantee the refcount + is > 1 going into the call. + +With the is\_unique instruction, the variable whose reference is being +checked for uniqueness appears mutable at the level of an individual SIL +instruction. After IRGen, is\_unique instructions are expanded into +runtime calls that no longer take the address of the variable. +Consequently, LLVM-level ARC optimization must be more conservative. It +must not remove retain/release pairs of this form: + + retain X + retain X + _swift_isUniquelyReferenced(X) + release X + release X + +To prevent removal of the apparently redundant inner retain/release +pair, the LLVM ARC optimizer should model \_swift\_isUniquelyReferenced +as a function that may release X, use X, and exit the program (the +subsequent release instruction does not prove safety). + +### is\_unique instruction + +As explained above, the SIL-level is\_unique instruction enforces the +semantics of uniqueness checks in the presence of ARC optimization. The +kind of reference count checking that is\_unique performs depends on the +argument type: + +> - Native object types are directly checked by reading the strong +> reference count: (Builtin.NativeObject, known native +> class reference) +> - Objective-C object types require an additional check that the +> dynamic object type uses native swift reference counting: +> (Builtin.UnknownObject, unknown class reference, +> class existential) +> - Bridged object types allow the dynamic object type check to be +> bypassed based on the pointer encoding: (Builtin.BridgeObject) + +Any of the above types may also be wrapped in an optional. If the static +argument type is optional, then a null check is also performed. + +Thus, is\_unique only returns true for non-null, native swift object +references with a strong reference count of one. + +is\_unique\_or\_pinned has the same semantics as is\_unique except that +it also returns true if the object is marked pinned (by strong\_pin) +regardless of the reference count. This allows for simultaneous +non-structural modification of multiple subobjects. + +### Builtin.isUnique + +Builtin.isUnique and Builtin.isUniqueOrPinned give the standard library +access to optimization safe uniqueness checking. Because the type of +reference check is derived from the builtin argument's static type, the +most efficient check is automatically generated. However, in some cases, +the standard library can dynamically determine that it has a native +reference even though the static type is a bridge or unknown object. +Unsafe variants of the builtin are available to allow the additional +pointer bit mask and dynamic class lookup to be bypassed in these cases: + +- isUnique\_native : <T> (inout T\[?\]) -> Int1 +- isUniqueOrPinned\_native : <T> (inout T\[?\]) -> Int1 + +These builtins perform an implicit cast to NativeObject before checking +uniqueness. There’s no way at SIL level to cast the address of a +reference, so we need to encapsulate this operation as part of the +builtin. diff --git a/docs/ARCOptimization.rst b/docs/ARCOptimization.rst deleted file mode 100644 index 082589310a756..0000000000000 --- a/docs/ARCOptimization.rst +++ /dev/null @@ -1,231 +0,0 @@ -:orphan: - -========================== -ARC Optimization for Swift -========================== - -.. contents:: - -.. admonition:: TODO - - This is currently a place holder for design documentation on ARC - optimization. - -Reference Counting Instructions -=============================== - -- strong_retain -- strong_retain_autoreleased -- strong_release -- strong_retain_unowned -- unowned_retain -- unowned_release -- load_weak -- store_weak -- fix_lifetime -- mark_dependence -- is_unique -- is_unique_or_pinned -- copy_block - -Memory Behavior of ARC Operations -================================= - -At SIL level, reference counting and reference checking instructions -are attributed with MayHaveSideEffects to prevent arbitrary passes -from reordering them. - -At IR level, retains are marked NoModRef with respect to load and -store instructions so they don't pessimize memory dependence. (Note -the Retains are still considered to write to memory with respect to -other calls because getModRefBehavior is not overridden.) Releases -cannot be marked NoModRef because they can have arbitrary side -effects. Is_unique calls cannot be marked NoModRef because they cannot -be reordered with other operations that may modify the reference -count. - -.. admonition:: TODO - - Marking runtime calls with NoModRef in LLVM is misleading (they - write memory), inconsistent (getModRefBehavior returns Unknown), - and fragile (e.g. if we inline ARC operations at IR level). To be - robust and allow stronger optimization, TBAA tags should be used to - indicate functions that only access object metadata. This would - also enable more LLVM level optimization in the presence of - is_unique checks which currently appear to arbitrarily write memory. - -RC Identity -=========== - -A core ARC concept in Swift optimization is the concept of ``Reference Count -Identity`` (RC Identity) and RC Identity preserving instructions. An instruction -``I`` with n SSA arguments and m SSA results is (i,j) RC Identity preserving if -performing a ``retain_value`` on the ith SSA argument immediately before ``I`` -is executed is equivalent to performing a ``retain_value`` on the jth SSA result -of ``I`` immediately following the execution of ``I``. For example in the -following, if:: - - retain_value %x - %y = unary_instruction %x - -is equivalent to:: - - %y = unary_instruction %x - retain_value %y - -then we say that unary_instruction is a (0,0) RC Identity preserving -operation. In a case of a unary instruction, we omit (0,0) and just say that -the instruction is RC Identity preserving. - -In practice generally RC Identical operations are unary operations such as -casts. This would make it seem like RC Identity is an extension of alias -analysis. But RC Identity also has significantly more power than alias analysis -since: - - - ``struct`` is an RC identity preserving operation if the ``struct`` literal - only has one non-trivial operand. This means for instance that any struct with - one reference counted field used as an owning pointer is RC Identical with its - owning pointer (a useful property for Arrays). - - - An ``enum`` instruction is always RC Identical with the given tuple payload. - - - A ``tuple`` instruction is an RC identity preserving operation if the - ``tuple`` literal has one non-trivial operand. - - - ``init_class_existential`` is an RC identity preserving operation since - performing a retain_value on a class existential is equivalent to performing - a retain_value on the class itself. - -The corresponding value projection operations have analogous properties. - -Given two SSA values ``%a``, ``%b``, we define ``%a`` as immediately RC -identical to ``%b`` if there exists an instruction ``I`` such that: - -- ``%a`` is the jth result of ``I``. -- ``%b`` is the ith argument of ``I``. -- ``I`` is (i,j) RC identity preserving. - -Easily the immediate RC identical relation must be reflexive and symmetric but -by its nature is not transitive. Then define the equivalence relation RC -Identity, ``~rc``, by the relations that ``%a ~rc %b`` if ``%a`` is immediately -RC identical to ``%b`` or if there is a finite sequence of n SSA values -``{%a[i]}`` such that ``%a`` is immediately RC identical to ``%a[0]`` and ``%b`` -is immediately RC identical to ``%a[n]``. We currently always assume that each -equivalence class has one dominating definition. - -These equivalence classes consisting of chains of RC identical values are -computed via the SILAnalysis called ``RC Identity Analysis``. By performing ARC -optimization on RC Identical operations, our optimizations are able to operate -on the level of granularity that we actually care about, ignoring superficial -changes in SSA form that still yield manipulations of the same reference count. - -*NOTE* RCIdentityAnalysis is a flow insensitive analysis. Dataflow that needs to - be flow sensitive must handle phi nodes in the dataflow itself. - -*NOTE* An important consequence of RC Identity is that value types with only one -RCIdentity are a simple case for ARC optimization to handle. The ARC optimizer -relies on other optimizations like SROA, Function Signature Opts, and -SimplifyCFG (for block arguments) to try and eliminate cases where value types -have multiple reference counted subtypes. - -Copy-On-Write Considerations -============================ - -The copy-on-write capabilities of some data structures, such as Array -and Set, are efficiently implemented via Builtin.isUnique calls which -lower directly to is_unique instructions in SIL. - -The is_unique instruction takes the address of a reference, and -although it does not actually change the reference, the reference must -appear mutable to the optimizer. This forces the optimizer to preserve -a retain distinct from what’s required to maintain lifetime for any of -the reference's source-level copies, because the called function is -allowed to replace the reference, thereby releasing the -referent. Consider the following sequence of rules: - -(1) An operation taking the address of a variable is allowed to - replace the reference held by that variable. The fact that - is_unique will not actually replace it is opaque to the optimizer. - -(2) If the refcount is 1 when the reference is replaced, the referent - is deallocated. - -(3) A different source-level variable pointing at the same referent - must not be changed/invalidated by such a call. - -(4) If such a variable exists, the compiler must guarantee the - refcount is > 1 going into the call. - -With the is_unique instruction, the variable whose reference is being -checked for uniqueness appears mutable at the level of an individual -SIL instruction. After IRGen, is_unique instructions are expanded into -runtime calls that no longer take the address of the -variable. Consequently, LLVM-level ARC optimization must be more -conservative. It must not remove retain/release pairs of this form: - -:: - - retain X - retain X - _swift_isUniquelyReferenced(X) - release X - release X - -To prevent removal of the apparently redundant inner retain/release -pair, the LLVM ARC optimizer should model _swift_isUniquelyReferenced -as a function that may release X, use X, and exit the program (the -subsequent release instruction does not prove safety). - -.. _arcopts.is_unique: - -is_unique instruction ---------------------- - -As explained above, the SIL-level is_unique instruction enforces the -semantics of uniqueness checks in the presence of ARC -optimization. The kind of reference count checking that -is_unique performs depends on the argument type: - - - Native object types are directly checked by reading the strong - reference count: - (Builtin.NativeObject, known native class reference) - - - Objective-C object types require an additional check that the - dynamic object type uses native swift reference counting: - (Builtin.UnknownObject, unknown class reference, class existential) - - - Bridged object types allow the dynamic object type check to be - bypassed based on the pointer encoding: - (Builtin.BridgeObject) - -Any of the above types may also be wrapped in an optional. If the -static argument type is optional, then a null check is also performed. - -Thus, is_unique only returns true for non-null, native swift object -references with a strong reference count of one. - -is_unique_or_pinned has the same semantics as is_unique except that it -also returns true if the object is marked pinned (by strong_pin) -regardless of the reference count. This allows for simultaneous -non-structural modification of multiple subobjects. - -Builtin.isUnique ----------------- - -Builtin.isUnique and Builtin.isUniqueOrPinned give the standard -library access to optimization safe uniqueness checking. Because the -type of reference check is derived from the builtin argument's static -type, the most efficient check is automatically generated. However, in -some cases, the standard library can dynamically determine that it has -a native reference even though the static type is a bridge or unknown -object. Unsafe variants of the builtin are available to allow the -additional pointer bit mask and dynamic class lookup to be bypassed in -these cases: - -- isUnique_native : (inout T[?]) -> Int1 -- isUniqueOrPinned_native : (inout T[?]) -> Int1 - -These builtins perform an implicit cast to NativeObject before -checking uniqueness. There’s no way at SIL level to cast the address -of a reference, so we need to encapsulate this operation as part of -the builtin. diff --git a/docs/AccessControl.md b/docs/AccessControl.md new file mode 100644 index 0000000000000..ce4a5680bbd34 --- /dev/null +++ b/docs/AccessControl.md @@ -0,0 +1,246 @@ +Access Control +============== + +The general guiding principle of Swift access control: + +> **No entity can be defined in terms of another entity that has a lower +> access level.** + +There are three levels of access: "private", "internal", and "public". +Private entities can only be accessed from within the source file where +they are defined. Internal entities can be accessed anywhere within the +module they are defined. Public entities can be accessed from anywhere +within the module and from any other context that imports the current +module. + +The names `public` and `private` have precedent in many languages; +`internal` comes from C\#. In the future, `public` may be used for both +API and SPI, at which point we may design additional annotations to +distinguish the two. + +By default, most entities in a source file have `internal` access. This +optimizes for the most common case—a single-target application +project—while not accidentally revealing entities to clients of a +framework module. + +Rules +----- + +Access to a particular entity is considered relative to the current +*access context.* The access context of an entity is the current file +(if `private`), the current module (if `internal`), or the current +program (if `public`). A reference to an entity may only be written +within the entity's access context. + +If a particular entity is not accessible, it does not appear in name +lookup, unlike in C++. However, access control does not restrict access +to members via runtime reflection (where applicable), nor does it +necessarily restrict visibility of symbols in a linked binary. + +### Globals and Members + +A global function, constant, or variable may have any access level less +than or equal to the access level of its type. That is, a `private` +constant can have `public` type, but not the other way around. + +Accessors for variables have the same access level as their associated +variable. The setter may be explicitly annotated with an access level +less than or equal to the access level of the variable; this is written +as `private(set)` or `internal(set)` before the `var` introducer. + +An initializer, method, subscript, or property may have any access level +less than or equal to the access level of its type (including the +implicit 'Self' type), with a few additional rules: + +- If the type's access level is `private`, the access level of members + defaults to `private`. If the type's access level is `internal` or + `public`, the access level of members defaults to `internal`. +- If a member is used to satisfy a protocol requirement, its access + level must be at least as high as the protocol conformance's; see + Protocols below. +- If an initializer is `required` by a superclass, its access level + must be at least as high as the access level of the subclass itself. +- Accessors for subscripts follow the same rules as accessors + for variables. +- A member may be overridden whenever it is accessible. + +The implicit memberwise initializer for a struct has the minimum access +level of all of the struct's stored properties, except that if all +properties are `public` the initializer is `internal`. The implicit +no-argument initializer for structs and classes follows the default +access level for the type. + +Currently, enum cases always have the same access level as the enclosing +enum. + +Deinitializers are only invoked by the runtime and do not nominally have +access. Internally, the compiler represents them as having the same +access level as the enclosing type. + +### Protocols + +A protocol may have any access level less than or equal to the access +levels of the protocols it refines. That is, a `private` ExtendedWidget +protocol can refine an `public` Widget protocol, but not the other way +around. + +The access level of a requirement is the access level of the enclosing +protocol, even when the protocol is `public`. Currently, requirements +may not be given a lower access level than the enclosing protocol. + +Swift does not currently support private protocol conformances, so for +runtime consistency, the access level of the conformance of type T to +protocol P is equal to the minimum of T's access level and P's access +level; that is, the conformance is accessible whenever both T and P are +accessible. This does not change if the protocol is conformed to in an +extension. (The access level of a conformance is not currently reflected +in the source, but is a useful concept for applying restrictions +consistently.) + +All members used to satisfy a conformance must have an access level at +least as high as the conformance's. This ensures consistency between +views of the type; if any member has a *lower* access level than the +conformance, then the member could be accessed anyway through a generic +function constrained by the protocol. + +> **note** +> +> This rule disallows an `internal` member of a protocol extension to +> satisfy a `public` requirement for a `public` type. Removing this +> limitation is not inherently unsafe, but (a) may be unexpected given +> the lack of explicit reference to the member, and (b) results in +> references to non-public symbols in the current representation. + +A protocol may be used as a type whenever it is accessible. A nominal +can conform to a protocol whenever the protocol is accessible. + +### Structs, Enums, and Classes + +A struct, enum, or class may be used as a type whenever it is +accessible. A struct, enum, or class may be extended whenever it is +accessible. + +A class may be subclassed whenever it is accessible. A class may have +any access level less than or equal to the access level of its +superclass. + +Members in an extension have the same default access level as members +declared within the extended type. However, an extension may be marked +with an explicit access modifier (e.g. `private extension`), in which +case the default access level of members within the extension is changed +to match. + +Extensions with explicit access modifiers may not add new protocol +conformances, since Swift does not support private protocol conformances +(see Protocols above). + +A type may conform to a protocol with lower access than the type itself. + +### Types + +A nominal type's access level is the same as the access level of the +nominal declaration itself. A generic type's access level is the minimum +of the access level of the base type and the access levels of all +generic argument types. + +A tuple type's access level is the minimum of the access levels of its +elements. A function type's access level is the minimum of the access +levels of its input and return types. + +A typealias may have any access level up to the access level of the type +it aliases. That is, a `private` typealias can refer to an `public` +type, but not the other way around. This includes associated types used +to satisfy protocol conformances. + +Runtime Guarantees +------------------ + +Non-`public` members of a class or extension will not be seen by +subclasses or other extensions from outside the module. Therefore, +members of a subclass or extension will not conflict with or +inadvertently be considered to override non-accessible members of the +superclass. + +Both `private` and `internal` increase opportunities for +devirtualization, though it is still possible to put a subclass of a +`private` class within the same file. + +Most information about a non-`public` entity still has to be put into a +module file for now, since we don't have resilience implemented. This +can be improved later, and is no more revealing than the information +currently available in the runtime for pure Objective-C classes. + +### Interaction with Objective-C + +If an entity is exposed to Objective-C, most of the runtime guarantees +and optimization opportunities go out the window. We have to use a +particular selector for members, everything can be inspected at runtime, +and even a private member can cause selector conflicts. In this case, +access control is only useful for discipline purposes. + +Members explicitly marked `private` are *not* exposed to Objective-C +unless they are also marked `@objc` (or `@IBAction` or similar), even if +declared within a class implicitly or explicitly marked `@objc`. + +Any `public` entities will be included in the generated header. In an +application or unit test target, `internal` entities will be exposed as +well. + +Non-Goals: "class-only" and "protected" +--------------------------------------- + +This proposal omits two forms of access control commonly found in other +languages, a "class-implementation-only" access (often called +"private"), and a "class and any subclasses" access (often called +"protected"). We chose not to include these levels of access control +because they do not add useful functionality beyond `private`, +`internal`, and `public`. + +"class-only" + +: If "class-only" includes extensions of the class, it is clear that + it provides no protection at all, since a class may be extended from + any context where it is accessible. So a hypothetical "class-only" + must already be limited with regards to extensions. Beyond that, + however, a "class-only" limit forces code to be declared within the + class that might otherwise naturally be a top-level helper or an + extension method on another type. + + `private` serves the proper use case of limiting access to the + implementation details of a class (even from the rest of + the module!) while not requiring that all of those implementation + details be written lexically inside the class. + +"protected" + +: "protected" access provides no guarantees of information hiding, + since any subclass can now access the implementation details of its + superclass---and expose them publicly, if it so chooses. This + interacts poorly with our future plans for resilient APIs. + Additionally, it increases the complexity of the access control + model for both the compiler and for developers, and like + "class-only" it is not immediately clear how it interacts + with extensions. + + Though it is not compiler-enforced, members that might be considered + "protected" are effectively publicly accessible, and thus should be + marked `public` in Swift. They can still be documented as intended + for overriding rather than for subclassing, but the specific details + of this are best dealt with on a case-by-case basis. + +Potential Future Directions +--------------------------- + +- Allowing `private` or `internal` protocol conformances, which are + only accessible at compile-time from a particular access context. +- Limiting particular capabilities, such as marking something + `final(public)` to restrict subclassing or overriding outside of the + current module. +- Allowing the Swift parts of a mixed-source framework to access + private headers. +- Revealing `internal` Swift API in a mixed-source framework in a + second generated header. +- Levels of `public`, for example `public("SPI")`. +- Enum cases less accessible than the enum. +- Protocol requirements less accessible than the protocol. + diff --git a/docs/AccessControl.rst b/docs/AccessControl.rst deleted file mode 100644 index eb45010656ab8..0000000000000 --- a/docs/AccessControl.rst +++ /dev/null @@ -1,253 +0,0 @@ -============== -Access Control -============== - -The general guiding principle of Swift access control: - - **No entity can be defined in terms of another entity that has a lower - access level.** - -There are three levels of access: "private", "internal", and "public". -Private entities can only be accessed from within the source file where they -are defined. Internal entities can be accessed anywhere within the module they -are defined. Public entities can be accessed from anywhere within the module -and from any other context that imports the current module. - -The names ``public`` and ``private`` have precedent in many languages; -``internal`` comes from C#. In the future, ``public`` may be used for both API -and SPI, at which point we may design additional annotations to distinguish the -two. - -By default, most entities in a source file have ``internal`` access. -This optimizes for the most common case—a single-target application -project—while not accidentally revealing entities to clients of a framework -module. - -.. contents:: :local: - -Rules -====== - -Access to a particular entity is considered relative to the current -*access context.* The access context of an entity is the current -file (if ``private``), the current module (if ``internal``), or the current -program (if ``public``). A reference to an entity may only be written within -the entity's access context. - -If a particular entity is not accessible, it does not appear in name lookup, -unlike in C++. However, access control does not restrict access to members via -runtime reflection (where applicable), nor does it necessarily restrict -visibility of symbols in a linked binary. - - -Globals and Members -------------------- - -A global function, constant, or variable may have any access level less than -or equal to the access level of its type. That is, a ``private`` constant can -have ``public`` type, but not the other way around. - -Accessors for variables have the same access level as their associated variable. -The setter may be explicitly annotated with an access level less than or equal -to the access level of the variable; this is written as ``private(set)`` or -``internal(set)`` before the ``var`` introducer. - -An initializer, method, subscript, or property may have any access level less -than or equal to the access level of its type (including the implicit 'Self' -type), with a few additional rules: - -- If the type's access level is ``private``, the access level of members - defaults to ``private``. If the type's access level is ``internal`` or - ``public``, the access level of members defaults to ``internal``. - -- If a member is used to satisfy a protocol requirement, its access level must - be at least as high as the protocol conformance's; see :ref:`Protocols` below. - -- If an initializer is ``required`` by a superclass, its access level must be - at least as high as the access level of the subclass itself. - -- Accessors for subscripts follow the same rules as accessors for variables. - -- A member may be overridden whenever it is accessible. - -The implicit memberwise initializer for a struct has the minimum access level -of all of the struct's stored properties, except that if all properties are -``public`` the initializer is ``internal``. The implicit no-argument -initializer for structs and classes follows the default access level for the -type. - -Currently, enum cases always have the same access level as the enclosing enum. - -Deinitializers are only invoked by the runtime and do not nominally have access. -Internally, the compiler represents them as having the same access level as the -enclosing type. - - -.. _Protocols: - -Protocols ---------- - -A protocol may have any access level less than or equal to the access levels -of the protocols it refines. That is, a ``private`` ExtendedWidget protocol can -refine an ``public`` Widget protocol, but not the other way around. - -The access level of a requirement is the access level of the enclosing -protocol, even when the protocol is ``public``. Currently, requirements may not -be given a lower access level than the enclosing protocol. - -Swift does not currently support private protocol conformances, so for runtime -consistency, the access level of the conformance of type T to protocol P is -equal to the minimum of T's access level and P's access level; that is, the -conformance is accessible whenever both T and P are accessible. This does not -change if the protocol is conformed to in an extension. (The access level of a -conformance is not currently reflected in the source, but is a useful concept -for applying restrictions consistently.) - -All members used to satisfy a conformance must have an access level at least as -high as the conformance's. This ensures consistency between views of the type; -if any member has a *lower* access level than the conformance, then the member -could be accessed anyway through a generic function constrained by the protocol. - -.. note:: - - This rule disallows an ``internal`` member of a protocol extension to satisfy - a ``public`` requirement for a ``public`` type. Removing this limitation is - not inherently unsafe, but (a) may be unexpected given the lack of explicit - reference to the member, and (b) results in references to non-public symbols - in the current representation. - -A protocol may be used as a type whenever it is accessible. A nominal can -conform to a protocol whenever the protocol is accessible. - - -Structs, Enums, and Classes ---------------------------- - -A struct, enum, or class may be used as a type whenever it is accessible. A -struct, enum, or class may be extended whenever it is accessible. - -A class may be subclassed whenever it is accessible. A class may have any -access level less than or equal to the access level of its superclass. - -Members in an extension have the same default access level as members declared -within the extended type. However, an extension may be marked with an explicit -access modifier (e.g. ``private extension``), in which case the default -access level of members within the extension is changed to match. - -Extensions with explicit access modifiers may not add new protocol -conformances, since Swift does not support private protocol conformances -(see :ref:`Protocols` above). - -A type may conform to a protocol with lower access than the type itself. - - -Types ------ - -A nominal type's access level is the same as the access level of the nominal -declaration itself. A generic type's access level is the minimum of the access -level of the base type and the access levels of all generic argument types. - -A tuple type's access level is the minimum of the access levels of its -elements. A function type's access level is the minimum of the access levels of -its input and return types. - -A typealias may have any access level up to the access level of the type it -aliases. That is, a ``private`` typealias can refer to an ``public`` type, but -not the other way around. This includes associated types used to satisfy -protocol conformances. - - -Runtime Guarantees -================== - -Non-``public`` members of a class or extension will not be seen by subclasses -or other extensions from outside the module. Therefore, members of a subclass -or extension will not conflict with or inadvertently be considered to override -non-accessible members of the superclass. - -Both ``private`` and ``internal`` increase opportunities for devirtualization, -though it is still possible to put a subclass of a ``private`` class within the -same file. - -Most information about a non-``public`` entity still has to be put into a -module file for now, since we don't have resilience implemented. This can be -improved later, and is no more revealing than the information currently -available in the runtime for pure Objective-C classes. - - -Interaction with Objective-C ----------------------------- - -If an entity is exposed to Objective-C, most of the runtime guarantees and -optimization opportunities go out the window. We have to use a particular -selector for members, everything can be inspected at runtime, and even a -private member can cause selector conflicts. In this case, access control is -only useful for discipline purposes. - -Members explicitly marked ``private`` are *not* exposed to Objective-C unless -they are also marked ``@objc`` (or ``@IBAction`` or similar), even if declared -within a class implicitly or explicitly marked ``@objc``. - -Any ``public`` entities will be included in the generated header. In an -application or unit test target, ``internal`` entities will be exposed as well. - - -Non-Goals: "class-only" and "protected" -======================================= - -This proposal omits two forms of access control commonly found in other -languages, a "class-implementation-only" access (often called "private"), and a -"class and any subclasses" access (often called "protected"). We chose not to -include these levels of access control because they do not add useful -functionality beyond ``private``, ``internal``, and ``public``. - -"class-only" - If "class-only" includes extensions of the class, it is clear that it - provides no protection at all, since a class may be extended from any context - where it is accessible. So a hypothetical "class-only" must already be - limited with regards to extensions. Beyond that, however, a "class-only" - limit forces code to be declared within the class that might otherwise - naturally be a top-level helper or an extension method on another type. - - ``private`` serves the proper use case of limiting access to the - implementation details of a class (even from the rest of the module!) while - not requiring that all of those implementation details be written lexically - inside the class. - -"protected" - "protected" access provides no guarantees of information hiding, since any - subclass can now access the implementation details of its superclass---and - expose them publicly, if it so chooses. This interacts poorly with our future - plans for resilient APIs. Additionally, it increases the complexity of the - access control model for both the compiler and for developers, and like - "class-only" it is not immediately clear how it interacts with extensions. - - Though it is not compiler-enforced, members that might be considered - "protected" are effectively publicly accessible, and thus should be marked - ``public`` in Swift. They can still be documented as intended for overriding - rather than for subclassing, but the specific details of this are best dealt - with on a case-by-case basis. - - -Potential Future Directions -=========================== - -- Allowing ``private`` or ``internal`` protocol conformances, which are only - accessible at compile-time from a particular access context. - -- Limiting particular capabilities, such as marking something ``final(public)`` - to restrict subclassing or overriding outside of the current module. - -- Allowing the Swift parts of a mixed-source framework to access private - headers. - -- Revealing ``internal`` Swift API in a mixed-source framework in a second - generated header. - -- Levels of ``public``, for example ``public("SPI")``. - -- Enum cases less accessible than the enum. - -- Protocol requirements less accessible than the protocol. diff --git a/docs/AccessControlInStdlib.md b/docs/AccessControlInStdlib.md new file mode 100644 index 0000000000000..7b5a3bb23d83f --- /dev/null +++ b/docs/AccessControlInStdlib.md @@ -0,0 +1,111 @@ +Scope and introduction +====================== + +This document defines the policy for applying access control modifiers +and related naming conventions for the Swift standard library and +overlays. + +In this document, “stdlib” refers to the core standard library and +overlays for system frameworks written in Swift. + +Swift has three levels of access control --- private, internal and +public. As currently implemented, access control is only concerned with +API-level issues, not ABI. The stdlib does not have a stable ABI, and is +compiled in "non-resilient" mode with inlining into user code; thus, all +stdlib symbols are considered ABI and stdlib clients should be +recompiled after *any* change to the stdlib. + +public +====== + +User-visible APIs should be marked public. + +Unfortunately, the compiler has bugs and limitations that the stdlib +must work around by defining additional public symbols not intended for +direct consumption by users. For example: + +These symbols are hidden using the leading underscore rule\_. + +Because Swift does not yet support a notion of SPI, any implementation +details that are shared across the stdlib's various sub-modules must +also be public. These names, too, use the leading underscore rule\_. + +To document the reason for marking symbols public, we use comments: + +- symbols used in tests: + + public // @testable + func _foo() { ... } + +- symbols that are SPIs for the module X: + + public // SPI(X) + public _foo() { ... } + +internal +======== + +In Swift, internal is an implied default everywhere—except within public +extensions and protocols. Therefore, internal should be used explicitly +everywhere in the stdlib to avoid confusion. + +> **note** +> +> No declaration should omit an access + +To create a “single point of truth” about whether a name is intended for +user consumption, the following names should all use the leading +underscore rule\_: + +- module-scope private and internal symbols: + + var _internalStdlibConstant: Int { ... } + +- private and internal symbols nested within public types: + + public struct Dictionary { + var _representation: _DictionaryRepresentation + } + +private +======= + +The private modifier can not be used in the stdlib at least until +rdar://17631278 is fixed. + +Leading Underscore Rule +======================= + +Variables, functions and typealiases should have names that start with +an underscore: + + var _value: Int + func _bridgeSomethingToAnything(something: AnyObject) -> AnyObject + typealias _InternalTypealias = HeapBuffer + +To apply the rule to an initializer, one of its label arguments *or* +internal parameter names must start with an underscore: + + public struct Foo { + init(_count: Int) {} + init(_ _otherInitializer: Int) {} + } + +> **note** +> +> the identifier that consists of a single underscore `_` is not + +> considered to be a name that starts with an underscore. For example, +> this initializer is public: +> +> public struct Foo { +> init(_ count: Int) {} +> } + +The compiler and IDE tools may use the leading underscore rule, combined +with additional heuristics, to hide stdlib symbols that users don't need +to see. + +Users are prohibited to use leading underscores symbols in their own +source code, even if these symbols are visible through compiler +diagnostics or IDE tools. diff --git a/docs/AccessControlInStdlib.rst b/docs/AccessControlInStdlib.rst deleted file mode 100644 index 20f857bba48d7..0000000000000 --- a/docs/AccessControlInStdlib.rst +++ /dev/null @@ -1,116 +0,0 @@ -:orphan: - -Scope and introduction -====================== - -This document defines the policy for applying access control modifiers and -related naming conventions for the Swift standard library and overlays. - -In this document, “stdlib” refers to the core standard library and -overlays for system frameworks written in Swift. - -Swift has three levels of access control --- private, internal -and public. As currently implemented, access control is only -concerned with API-level issues, not ABI. The stdlib does not have a stable ABI, -and is compiled in "non-resilient" mode with inlining into user code; thus, all -stdlib symbols are considered ABI and stdlib clients should be recompiled after -*any* change to the stdlib. - -`public` -======== - -User-visible APIs should be marked public. - -Unfortunately, the compiler has bugs and limitations that the stdlib -must work around by defining additional public symbols not intended -for direct consumption by users. For example: - -.. parsed-literal:: - - // Workaround. - public protocol **_Collection** { ... } - - // Symbol intended for use outside stdlib. - public protocol Collection : **_Collection** { ... } - -These symbols are hidden using the `leading underscore rule`_. - -Because Swift does not yet support a notion of SPI, any implementation -details that are shared across the stdlib's various sub-modules must -also be public. These names, too, use the `leading underscore rule`_. - -To document the reason for marking symbols public, we use comments: - -* symbols used in tests:: - - public // @testable - func _foo() { ... } - -* symbols that are SPIs for the module X:: - - public // SPI(X) - public _foo() { ... } - -`internal` -========== - -In Swift, `internal` is an implied default everywhere—except within -`public` extensions and protocols. Therefore, `internal` should be used -explicitly everywhere in the stdlib to avoid confusion. - -.. Note:: No declaration should omit an access - -To create a “single point of truth” about whether a name is intended -for user consumption, the following names should all use the `leading -underscore rule`_: - -* module-scope `private` and `internal` symbols:: - - var _internalStdlibConstant: Int { ... } - -* `private` and `internal` symbols nested within `public` types:: - - public struct Dictionary { - var _representation: _DictionaryRepresentation - } - -`private` -========= - -The `private` modifier can not be used in the stdlib at least until -rdar://17631278 is fixed. - -Leading Underscore Rule -======================= - -Variables, functions and typealiases should have names that start with an -underscore:: - - var _value: Int - func _bridgeSomethingToAnything(something: AnyObject) -> AnyObject - typealias _InternalTypealias = HeapBuffer - -To apply the rule to an initializer, one of its label arguments *or* -internal parameter names must start with an underscore:: - - public struct Foo { - init(_count: Int) {} - init(_ _otherInitializer: Int) {} - } - -.. Note:: the identifier that consists of a single underscore ``_`` is not - considered to be a name that starts with an underscore. For example, this - initializer is public:: - - public struct Foo { - init(_ count: Int) {} - } - -The compiler and IDE tools may use the leading underscore rule, -combined with additional heuristics, to hide stdlib symbols that users -don't need to see. - -Users are prohibited to use leading underscores symbols in their own source -code, even if these symbols are visible through compiler diagnostics -or IDE tools. - diff --git a/docs/Arrays.md b/docs/Arrays.md new file mode 100644 index 0000000000000..3f87d9c4f42f3 --- /dev/null +++ b/docs/Arrays.md @@ -0,0 +1,261 @@ +The Swift Array Design +====================== + +Author + +: Dave Abrahams + +Date + +: 2014-04-10 + + +Goals +----- + +1. Performance equivalent to C arrays for subscript get/set of + non-class element types is the most important performance goal. +2. It should be possible to receive an `NSArray` from Cocoa, represent + it as an `Array`, and pass it right back to Cocoa as an + `NSArray` in O(1) and with no memory allocations. +3. Arrays should be usable as stacks, so we want amortized O(1) append + and O(1) popBack. Together with goal \#1, this implies a + `std::vector`-like layout, with a reserved tail memory capacity that + can exceed the number of actual stored elements. + +To achieve goals 1 and 2 together, we use static knowledge of the +element type: when it is statically known that the element type is not a +class, code and checks accounting for the possibility of wrapping an +`NSArray` are eliminated. An `Array` of Swift value types always uses +the most efficient possible representation, identical to that of +`ContiguousArray`. + +Components +---------- + +Swift provides three generic array types, all of which have amortized +O(1) growth. In this document, statements about **ArrayType** apply to +all three of the components. + +- `ContiguousArray` is the fastest and simplest of the three—use + this when you need "C array" performance. The elements of a + `ContiguousArray` are always stored contiguously in memory. + + ![image](ContiguousArray.png) + +- `Array` is like `ContiguousArray`, but optimized for efficient + conversions from Cocoa and back—when `T` can be a class type, + `Array` can be backed by the (potentially non-contiguous) storage + of an arbitrary `NSArray` rather than by a Swift `ContiguousArray`. + `Array` also supports up- and down- casts between arrays of + related class types. When `T` is known to be a non-class type, the + performance of `Array` is identical to that of + `ContiguousArray`. + + ![image](ArrayImplementation.png) + +- `Slice` is a subrange of some `Array` or `ContiguousArray`; + it's the result of using slice notation, e.g. `a[7...21]` on any + Swift array `a`. A slice always has contiguous storage and "C + array" performance. Slicing an *ArrayType* is O(1) unless the source + is an `Array` backed by an `NSArray` that doesn't supply + contiguous storage. + + `Slice` is recommended for transient computations but not for + long-term storage. Since it references a sub-range of some shared + backing buffer, a `Slice` may artificially prolong the lifetime of + elements outside the `Slice` itself. + + ![image](Slice.png) + +Mutation Semantics +------------------ + +The *ArrayType*s have full value semantics via copy-on-write (COW): + + var a = [1, 2, 3] + let b = a + a[1] = 42 + print(b[1]) // prints "2" + +Bridging Rules and Terminology for all Types +-------------------------------------------- + +- Every class type or `@objc` existential (such as `AnyObject`) is + **bridged** to Objective-C and **bridged back** to Swift via the + identity transformation, i.e. it is **bridged verbatim**. +- A type `T` that is not bridged verbatim\_ can conform to + `BridgedToObjectiveC`, which specifies its conversions to and from + ObjectiveC: + + protocol _BridgedToObjectiveC { + typealias _ObjectiveCType: AnyObject + func _bridgeToObjectiveC() -> _ObjectiveCType + class func _forceBridgeFromObjectiveC(_: _ObjectiveCType) -> Self + } + + > **note** + > + > Classes and `@objc` existentials shall not conform to + > `_BridgedToObjectiveC`, a restriction that's not currently + > enforceable at compile-time. + +- Some generic types (*ArrayType*`` in particular) bridge to + Objective-C only if their element types bridge. These types conform + to `_ConditionallyBridgedToObjectiveC`: + + protocol _ConditionallyBridgedToObjectiveC : _BridgedToObjectiveC { + class func _isBridgedToObjectiveC() -> Bool + class func _conditionallyBridgeFromObjectiveC(_: _ObjectiveCType) -> Self? + } + + Bridging from, or *bridging back* to, a type `T` conforming to + `_ConditionallyBridgedToObjectiveC` when + `T._isBridgedToObjectiveC()` is `false` is a user programming error + that may be diagnosed at runtime. + `_conditionallyBridgeFromObjectiveC` can be used to attempt to + bridge back, and return `nil` if the entire object cannot + be bridged. + + > **Implementation Note** + > + > There are various ways to move this detection to compile-time + +- For a type `T` that is not bridged verbatim\_, + - if `T` conforms to `BridgedToObjectiveC` and either + + - `T` does not conform to `_ConditionallyBridgedToObjectiveC` + - or, `T._isBridgedToObjectiveC()` + + then a value `x` of type `T` is **bridged** as + `T._ObjectiveCType` via `x._bridgeToObjectiveC()`, and an object + `y` of `T._ObjectiveCType` is **bridged back** to `T` via + `T._forceBridgeFromObjectiveC(y)` + + - Otherwise, `T` **does not bridge** to Objective-C + +`Array` Type Conversions +------------------------ + +From here on, this document deals only with `Array` itself, and not +`Slice` or `ContiguousArray`, which support a subset of `Array` +'s conversions. Future revisions will add descriptions of `Slice` and +`ContiguousArray` conversions. + +### Kinds of Conversions + +In these definitions, `Base` is `AnyObject` or a trivial subtype +thereof, `Derived` is a trivial subtype of `Base`, and `X` conforms to +`_BridgedToObjectiveC`: + +- **Trivial bridging** implicitly converts `Base[]` to `NSArray` + in O(1). This is simply a matter of returning the Array's internal + buffer, which is-a `NSArray`. + + + +- **Trivial bridging back** implicitly converts `NSArray` to + `AnyObject[]` in O(1) plus the cost of calling `copy()` on the + `NSArray`. [^1] +- **Implicit conversions** between `Array` types + + - **Implicit upcasting** implicitly converts `Derived[]` to + `Base[]` in O(1). + - **Implicit bridging** implicitly converts `X[]` to + `X._ObjectiveCType[]` in O(N). + + > **note** + > + > Either type of implicit conversion may be combined with + > trivial bridging\_ in an implicit conversion to `NSArray`. + +- **Checked conversions** convert `T[]` to `U[]?` in O(N) via + `a as U[]`. + - **Checked downcasting** converts `Base[]` to `Derived[]?`. + - **Checked bridging back** converts `T[]` to `X[]?` where + `X._ObjectiveCType` is `T` or a trivial subtype thereof. +- **Forced conversions** convert `AnyObject[]` or `NSArray` to `T[]` + implicitly, in bridging thunks between Swift and Objective-C. + + For example, when a user writes a Swift method taking `NSView[]`, it + is exposed to Objective-C as a method taking `NSArray`, which is + force-converted to `NSView[]` when called from Objective-C. + + - **Forced downcasting** converts `AnyObject[]` to `Derived[]` + in O(1) + - **Forced bridging back** converts `AnyObject[]` to `X[]` + in O(N). + + A forced conversion where any element fails to convert is considered + a user programming error that may trap. In the case of forced + downcasts, the trap may be deferred\_ to the point where an + offending element is accessed. + +> **note** +> +> Both checked and forced downcasts may be combined with trivial +> bridging back\_ in conversions from `NSArray`. + +### Maintaining Type-Safety + +Both upcasts and forced downcasts raise type-safety issues. + +#### Upcasts + +TODO: this section is outdated. + +When up-casting an `Derived[]` to `Base[]`, a buffer of `Derived` object +can simply be `unsafeBitCast`'ed to a buffer of elements of type +`Base`—as long as the resulting buffer is never mutated. For example, we +cannot allow a `Base` element to be inserted in the buffer, because the +buffer's destructor will destroy the elements with the (incorrect) +static presumption that they have `Derived` type. + +Furthermore, we can't (logically) copy the buffer just prior to +mutation, since the `Base[]` may be copied prior to mutation, and our +shared subscript assignment semantics imply that all copies must observe +its subscript assignments. + +Therefore, converting `T[]` to `U[]` is akin to resizing: the new +`Array` becomes logically independent. To avoid an immediate O(N) +conversion cost, and preserve shared subscript assignment semantics, we +use a layer of indirection in the data structure. Further, when `T` is a +subclass of `U`, the intermediate object is marked to prevent in-place +mutation of the buffer; it will be copied upon its first mutation: + +![image](ArrayCast.png) + +#### Deferred Checking for Forced Downcasts + +In forced downcasts, if any element fails to have dynamic type +`Derived`, it is considered a programming error that may cause a trap. +Sometimes we can do this check in O(1) because the source holds a known +buffer type. Rather than incur O(N) checking for the other cases, the +new intermediate object is marked for deferred checking, and all element +accesses through that object are dynamically typechecked, with a trap +upon failure (except in `-Ounchecked` builds). + +When the resulting array is later up-cast (other than to a type that can +be validated in O(1) by checking the type of the underlying buffer), the +result is also marked for deferred checking. + +------------------------------------------------------------------------ + +[^1]: This `copy()` may amount to a retain if the `NSArray` is already + known to be immutable. We could eventually optimize out the copy if + we can detect that the `NSArray` is uniquely referenced. Our current + unique-reference detection applies only to Swift objects, though. diff --git a/docs/Arrays.rst b/docs/Arrays.rst deleted file mode 100644 index 766d30522b8bd..0000000000000 --- a/docs/Arrays.rst +++ /dev/null @@ -1,274 +0,0 @@ -:orphan: - -The Swift Array Design -====================== - -:Author: Dave Abrahams -:Date: 2014-04-10 - -.. raw:: html - - - -Goals ------ - -1. Performance equivalent to C arrays for subscript get/set of - non-class element types is the most important performance goal. - -2. It should be possible to receive an ``NSArray`` from Cocoa, - represent it as an ``Array``, and pass it right back to - Cocoa as an ``NSArray`` in O(1) and with no memory allocations. - -3. Arrays should be usable as stacks, so we want amortized O(1) append - and O(1) popBack. Together with goal #1, this implies a - ``std::vector``\ -like layout, with a reserved tail memory capacity - that can exceed the number of actual stored elements. - -To achieve goals 1 and 2 together, we use static knowledge of the -element type: when it is statically known that the element type is not -a class, code and checks accounting for the possibility of wrapping an -``NSArray`` are eliminated. An ``Array`` of Swift value types always -uses the most efficient possible representation, identical to that of -``ContiguousArray``. - -Components ----------- - -Swift provides three generic array types, all of which have amortized -O(1) growth. In this document, statements about **ArrayType** apply -to all three of the components. - -* ``ContiguousArray`` is the fastest and simplest of the three—use this - when you need "C array" performance. The elements of a - ``ContiguousArray`` are always stored contiguously in memory. - - .. image:: ContiguousArray.png - -* ``Array`` is like ``ContiguousArray``, but optimized for efficient - conversions from Cocoa and back—when ``T`` can be a class type, - ``Array`` can be backed by the (potentially non-contiguous) - storage of an arbitrary ``NSArray`` rather than by a Swift - ``ContiguousArray``. ``Array`` also supports up- and down- casts - between arrays of related class types. When ``T`` is known to be a - non-class type, the performance of ``Array`` is identical to that - of ``ContiguousArray``. - - .. image:: ArrayImplementation.png - -* ``Slice`` is a subrange of some ``Array`` or - ``ContiguousArray``; it's the result of using slice notation, - e.g. ``a[7...21]`` on any Swift array ``a``. A slice always has - contiguous storage and "C array" performance. Slicing an - *ArrayType* is O(1) unless the source is an ``Array`` backed by - an ``NSArray`` that doesn't supply contiguous storage. - - ``Slice`` is recommended for transient computations but not for - long-term storage. Since it references a sub-range of some shared - backing buffer, a ``Slice`` may artificially prolong the lifetime of - elements outside the ``Slice`` itself. - - .. image:: Slice.png - -Mutation Semantics ------------------- - -The *ArrayType*\ s have full value semantics via copy-on-write (COW):: - - var a = [1, 2, 3] - let b = a - a[1] = 42 - print(b[1]) // prints "2" - -Bridging Rules and Terminology for all Types --------------------------------------------- - -.. _bridged verbatim: - -* Every class type or ``@objc`` existential (such as ``AnyObject``) is - **bridged** to Objective-C and **bridged back** to Swift via the - identity transformation, i.e. it is **bridged verbatim**. - -* A type ``T`` that is not `bridged verbatim`_ can conform to - ``BridgedToObjectiveC``, which specifies its conversions to and from - ObjectiveC:: - - protocol _BridgedToObjectiveC { - typealias _ObjectiveCType: AnyObject - func _bridgeToObjectiveC() -> _ObjectiveCType - class func _forceBridgeFromObjectiveC(_: _ObjectiveCType) -> Self - } - - .. Note:: Classes and ``@objc`` existentials shall not conform to - ``_BridgedToObjectiveC``, a restriction that's not currently - enforceable at compile-time. - -* Some generic types (*ArrayType*\ ```` in particular) bridge to - Objective-C only if their element types bridge. These types conform - to ``_ConditionallyBridgedToObjectiveC``:: - - protocol _ConditionallyBridgedToObjectiveC : _BridgedToObjectiveC { - class func _isBridgedToObjectiveC() -> Bool - class func _conditionallyBridgeFromObjectiveC(_: _ObjectiveCType) -> Self? - } - - Bridging from, or *bridging back* to, a type ``T`` conforming to - ``_ConditionallyBridgedToObjectiveC`` when - ``T._isBridgedToObjectiveC()`` is ``false`` is a user programming - error that may be diagnosed at - runtime. ``_conditionallyBridgeFromObjectiveC`` can be used to attempt - to bridge back, and return ``nil`` if the entire object cannot be - bridged. - - .. Admonition:: Implementation Note - - There are various ways to move this detection to compile-time - -* For a type ``T`` that is not `bridged verbatim`_, - - - if ``T`` conforms to ``BridgedToObjectiveC`` and either - - - ``T`` does not conform to ``_ConditionallyBridgedToObjectiveC`` - - or, ``T._isBridgedToObjectiveC()`` - - then a value ``x`` of type ``T`` is **bridged** as - ``T._ObjectiveCType`` via ``x._bridgeToObjectiveC()``, and an object - ``y`` of ``T._ObjectiveCType`` is **bridged back** to ``T`` via - ``T._forceBridgeFromObjectiveC(y)`` - - - Otherwise, ``T`` **does not bridge** to Objective-C - -``Array`` Type Conversions --------------------------- - -From here on, this document deals only with ``Array`` itself, and not -``Slice`` or ``ContiguousArray``, which support a subset of ``Array``\ -'s conversions. Future revisions will add descriptions of ``Slice`` -and ``ContiguousArray`` conversions. - -Kinds of Conversions -:::::::::::::::::::: - -In these definitions, ``Base`` is ``AnyObject`` or a trivial subtype -thereof, ``Derived`` is a trivial subtype of ``Base``, and ``X`` -conforms to ``_BridgedToObjectiveC``: - -.. _trivial bridging: - -* **Trivial bridging** implicitly converts ``Base[]`` to - ``NSArray`` in O(1). This is simply a matter of returning the - Array's internal buffer, which is-a ``NSArray``. - -.. _trivial bridging back: - -* **Trivial bridging back** implicitly converts ``NSArray`` to - ``AnyObject[]`` in O(1) plus the cost of calling ``copy()`` on - the ``NSArray``. [#nocopy]_ - -* **Implicit conversions** between ``Array`` types - - - **Implicit upcasting** implicitly converts ``Derived[]`` to - ``Base[]`` in O(1). - - **Implicit bridging** implicitly converts ``X[]`` to - ``X._ObjectiveCType[]`` in O(N). - - .. Note:: Either type of implicit conversion may be combined with - `trivial bridging`_ in an implicit conversion to ``NSArray``. - -* **Checked conversions** convert ``T[]`` to ``U[]?`` in O(N) - via ``a as U[]``. - - - **Checked downcasting** converts ``Base[]`` to ``Derived[]?``. - - **Checked bridging back** converts ``T[]`` to ``X[]?`` where - ``X._ObjectiveCType`` is ``T`` or a trivial subtype thereof. - -* **Forced conversions** convert ``AnyObject[]`` or ``NSArray`` to - ``T[]`` implicitly, in bridging thunks between Swift and Objective-C. - - For example, when a user writes a Swift method taking ``NSView[]``, - it is exposed to Objective-C as a method taking ``NSArray``, which - is force-converted to ``NSView[]`` when called from Objective-C. - - - **Forced downcasting** converts ``AnyObject[]`` to ``Derived[]`` in - O(1) - - **Forced bridging back** converts ``AnyObject[]`` to ``X[]`` in O(N). - - A forced conversion where any element fails to convert is considered - a user programming error that may trap. In the case of forced - downcasts, the trap may be deferred_ to the point where an offending - element is accessed. - -.. Note:: Both checked and forced downcasts may be combined with `trivial - bridging back`_ in conversions from ``NSArray``. - -Maintaining Type-Safety -::::::::::::::::::::::: - -Both upcasts and forced downcasts raise type-safety issues. - -Upcasts -....... - -TODO: this section is outdated. - -When up-casting an ``Derived[]`` to ``Base[]``, a buffer of -``Derived`` object can simply be ``unsafeBitCast``\ 'ed to a buffer -of elements of type ``Base``—as long as the resulting buffer is never -mutated. For example, we cannot allow a ``Base`` element to be -inserted in the buffer, because the buffer's destructor will destroy -the elements with the (incorrect) static presumption that they have -``Derived`` type. - -Furthermore, we can't (logically) copy the buffer just prior to -mutation, since the ``Base[]`` may be copied prior to mutation, -and our shared subscript assignment semantics imply that all copies -must observe its subscript assignments. - -Therefore, converting ``T[]`` to ``U[]`` is akin to -resizing: the new ``Array`` becomes logically independent. To avoid -an immediate O(N) conversion cost, and preserve shared subscript -assignment semantics, we use a layer of indirection in the data -structure. Further, when ``T`` is a subclass of ``U``, the -intermediate object is marked to prevent in-place mutation of the -buffer; it will be copied upon its first mutation: - -.. image:: ArrayCast.png - -.. _deferred: - -Deferred Checking for Forced Downcasts -....................................... - -In forced downcasts, if any element fails to have dynamic type ``Derived``, -it is considered a programming error that may cause a trap. Sometimes -we can do this check in O(1) because the source holds a known buffer -type. Rather than incur O(N) checking for the other cases, the new -intermediate object is marked for deferred checking, and all element -accesses through that object are dynamically typechecked, with a trap -upon failure (except in ``-Ounchecked`` builds). - -When the resulting array is later up-cast (other than to a type that -can be validated in O(1) by checking the type of the underlying -buffer), the result is also marked for deferred checking. - ----- - -.. [#nocopy] This ``copy()`` may amount to a retain if the ``NSArray`` - is already known to be immutable. We could eventually optimize out - the copy if we can detect that the ``NSArray`` is uniquely - referenced. Our current unique-reference detection applies only to - Swift objects, though. diff --git a/docs/CallingConvention.md b/docs/CallingConvention.md new file mode 100644 index 0000000000000..08b4ca3cf9823 --- /dev/null +++ b/docs/CallingConvention.md @@ -0,0 +1,1157 @@ +The Swift Calling Convention +============================ + +This whitepaper discusses the Swift calling convention, at least as we +want it to be. + +It's a basic assumption in this paper that Swift shouldn't make an +implicit promise to exactly match the default platform calling +convention. That is, if a C or Objective-C programmer manages to derive +the address of a Swift function, we don't have to promise that an +obvious translation of the type of that function will be correctly +callable from C. For example, this wouldn't be guaranteed to work: + + // In Swift: + func foo(x: Int, y: Double) -> MyClass { ... } + + // In Objective-C: + extern id _TF4main3fooFTSiSd_CS_7MyClass(intptr_t x, double y); + +We do sometimes need to be able to match C conventions, both to use them +and to generate implementations of them, but that level of compatibility +should be opt-in and site-specific. If Swift would benefit from +internally using a better convention than C/Objective-C uses, and +switching to that convention doesn't damage the dynamic abilities of our +target platforms (debugging, dtrace, stack traces, unwinding, etc.), +there should be nothing preventing us from doing so. (If we did want to +guarantee compatibility on this level, this paper would be a lot +shorter!) + +Function call rules in high-level languages have three major components, +each operating on a different abstraction level: + +- the high-level semantics of the call (pass-by-reference vs. + pass-by-value), +- the ownership and validity conventions about argument and result + values ("+0" vs. "+1", etc.), and +- the "physical" representation conventions of how values are actually + communicated between functions (in registers, on the stack, etc.). + +We'll tackle each of these in turn, then conclude with a detailed +discussion of function signature lowering. + +High-level semantic conventions +------------------------------- + +The major division in argument passing conventions between languages is +between pass-by-reference and pass-by-value languages. It's a +distinction that only really makes sense in languages with the concept +of an l-value, but Swift does, so it's pertinent. + +In general, the terms "pass-by-X" and "call-by-X" are used +interchangeably. It's unfortunate. We'll prefer "pass-by-X" for +consistency and to emphasize that these conventions are +argument-specific. + +### Pass-by-reference + +In pass-by-reference (also called pass-by-name or pass-by-address), if A +is an l-value expression, foo(A) is passed some sort of opaque reference +through which the original l-value can be modified. If A is not an +l-value, the language may prohibit this, or (if pass-by-reference is the +default convention) it may pass a temporary variable containing the +result of A. + +Don't confuse pass-by-reference with the concept of a *reference type*. +A reference type is a type whose value is a reference to a different +object; for example, a pointer type in C, or a class type in Java or +Swift. A variable of reference type can be passed by value (copying the +reference itself) or by reference (passing the variable itself, allowing +it to be changed to refer to a different object). Note that references +in C++ are a generalization of pass-by-reference, not really a reference +type; in C++, a variable of reference type behaves completely unlike any +other variable in the language. + +Also, don't confuse pass-by-reference with the physical convention of +passing an argument value indirectly. In pass-by-reference, what's +logically being passed is a reference to a tangible, user-accessible +object; changes to the original object will be visible in the reference, +and changes to the reference will be reflected in the original object. +In an indirect physical convention, the argument is still logically an +independent value, no longer associated with the original object (if +there was one). + +If every object in the language is stored in addressable memory, +pass-by-reference can be easily implemented by simply passing the +address of the object. If an l-value can have more structure than just a +single, independently-addressable object, more information may be +required from the caller. For example, an array argument in FORTRAN can +be a row or column vector from a matrix, and so arrays are generally +passed as both an address and a stride. C and C++ do have unaddressable +l-values because of bitfields, but they forbid passing bitfields by +reference (in C++) or taking their address (in either language), which +greatly simplifies pointer and reference types in those languages. + +FORTRAN is the last remaining example of a language that defaults to +pass-by-reference. Early FORTRAN implementations famously passed +constants by passing the address of mutable global memory initialized to +the constant; if the callee modified its parameter (illegal under the +standard, but...), it literally changed the constant for future uses. +FORTRAN now allows procedures to explicitly take arguments by value and +explicitly declare that arguments must be l-values. + +However, many languages do allow parameters to be explicitly marked as +pass-by-reference. As mentioned for C++, sometimes only certain kinds of +l-values are allowed. + +Swift allows parameters to be marked as pass-by-reference with inout. +Arbitrary l-values can be passed. The Swift convention is to always pass +an address; if the parameter is not addressable, it must be materialized +into a temporary and then written back. See the accessors proposal for +more details about the high-level semantics of inout arguments. + +### Pass-by-value + +In pass-by-value, if A is an l-value expression, foo(A) copies the +current value there. Any modifications foo makes to its parameter are +made to this copy, not to the original l-value. + +Most modern languages are pass-by-value, with specific functions able to +opt in to pass-by-reference semantics. This is exactly what Swift does. + +There's not much room for variation in the high-level semantics of +passing arguments by value; all the variation is in the ownership and +physical conventions. + +Ownership transfer conventions +------------------------------ + +Arguments and results that require cleanup, like an Objective-C object +reference or a non-POD C++ object, raise two questions about +responsibility: who is responsible for cleaning it up, and when? + +These questions arise even when the cleanup is explicit in code. C's +strdup function returns newly-allocated memory which the caller is +responsible for freeing, but strtok does not. Objective-C has standard +naming conventions that describe which functions return objects that the +caller is responsible for releasing, and outside of ARC these must be +followed manually. Of course, conventions designed to be implemented by +programmers are often designed around the simplicity of that +implementation, rather than necessarily being more efficient. + +### Pass-by-reference arguments + +Pass-by-reference arguments generally don't involve a *transfer* of +ownership. It's assumed that the caller will ensure that the referent is +valid at the time of the call, and that the callee will ensure that the +referent is still valid at the time of return. + +FORTRAN does actually allow parameters to be tagged as out-parameters, +where the caller doesn't guarantee the validity of the argument before +the call. Objective-C has something similar, where an indirect method +argument can be marked out; ARC takes advantage of this with +autoreleasing parameters to avoid a copy into the writeback temporary. +Neither of these are something we semantically care about supporting in +Swift. + +There is one other theoretically interesting convention question here: +the argument has to be valid before the call and after the call, but +does it have to valid during the call? Swift's answer to this is +generally "yes". Swift does have inout aliasing rules that allow a +certain amount of optimization, but the compiler is forbidden from +exploiting these rules in any way that could cause memory corruption (at +least in the absence of race conditions). So Swift has to ensure that an +inout argument is valid whenever it does something (including calling an +opaque function) that could potentially access the original l-value. + +If Swift allowed local variables to be captured through inout +parameters, and therefore needed to pass an implicit owner parameter +along with an address, this owner parameter would behave like a +pass-by-value argument and could use any of the conventions listed +below. However, the optimal convention for this is obvious: it should be +guaranteed, since captures are very unlikely and callers are almost +always expected to use the value of an inout variable afterwards. + +### Pass-by-value arguments + +All conventions for this have performance trade-offs. + +We're only going to discuss *static* conventions, where the transfer is +picked at compile time. It's possible to have a *dynamic* convention, +where the caller passes a flag indicating whether it's okay to directly +take responsibility for the value, and the callee can (conceptually) +return a flag indicating whether it actually did take responsibility for +it. If copying is extremely expensive, that can be worthwhile; +otherwise, the code cost may overwhelm any other benefits. + +This discussion will ignore one particular impact of these conventions +on code size. If a function has many callers, conventions that require +more code in the caller are worse, all else aside. If a single call site +has many possible targets, conventions that require more code in the +callee are worse, all else aside. It's not really reasonable to decide +this in advance for unknown code; we could maybe make rules about code +calling system APIs, except that system APIs are by definition locked +down, and we can't change them. It's a reasonable thing to consider +changing with PGO, though. + +#### Responsibility + +A common refrain in this performance analysis will be whether a function +has responsibility for a value. A function has to get a value from +*somewhere*: + +- A caller is usually responsible for the return values it receives: + the callee generated the value and the caller is responsible for + destroying it. Any other convention has to rely on heavily + restricting what kind of value can be returned. (If you're thinking + about Objective-C autoreleased results, just accept this for now; + we'll talk about that later.) +- A function isn't necessarily responsible for a value it loads + from memory. Ignoring race conditions, the function may be able to + immediately use the value without taking any specific action to keep + it valid. +- A callee may or may not be responsible for a value passed as a + parameter, depending on the convention it was passed with. +- A function might come from a source that doesn't necessarily make + the function responsible, but if the function takes an action which + invalidates the source before using the value, the function has to + take action to keep the value valid. At that point, the function has + responsibility for the value despite its original source. + + For example, a function foo() might load a reference r from a global + variable x, call an unknown function bar(), and then use r in + some way. If bar() can't possibly overwrite x, foo() doesn't have to + do anything to keep r alive across the call; otherwise it does (e.g. + by retaining it in a refcounted environment). This is a situation + where humans are often much smarter than compilers. Of course, it's + also a situation where humans are sometimes + insufficiently conservative. + +A function may also require responsibility for a value as part of its +operation: + +- Since a variable is always responsible for the current value it + stores, a function which stores a value into memory must first gain + responsibility for that value. +- A callee normally transfers responsibility for its return value to + its caller; therefore it must gain responsibility for its return + value before returning it. +- A caller may need to gain responsibility for a value before passing + it as an argument, depending on the parameter's + ownership-transfer convention. + +#### Known conventions + +There are three static parameter conventions for ownership worth +considering here: + +- The caller may transfer responsibility for the value to the callee. + In SIL, we call this an **owned** parameter. + + This is optimal if the caller has responsibility for the value and + doesn't need it after the call. This is an extremely common + situation; for example, it comes up whenever a call result is + immediately used an argument. By giving the callee responsibility + for the value, this convention allows the callee to use the value at + a later point without taking any extra action to keep it alive. + + The flip side is that this convention requires a lot of extra work + when a single value is used multiple times in the caller. For + example, a value passed in every iteration of a loop will need to be + copied/retained/whatever each time. + +- The caller may provide the value without any responsibility on + either side. In SIL, we call this an **unowned** parameter. The + value is guaranteed to be valid at the moment of the call, and in + the absence of race conditions, that guarantee can be assumed to + continue unless the callee does something that might invalidate it. + As discussed above, humans are often much smarter than computers + about knowing when that's possible. + + This is optimal if the caller can acquire the value without + responsibility and the callee doesn't require responsibility of it. + In very simple code --- e.g., loading values from an array and + passing them to a comparator function which just reads a few fields + from each and returns --- this can be extremely efficient. + + Unfortunately, this convention is completely undermined if either + side has to do anything that forces it to take action to keep the + value alive. Also, if that happens on the caller side, the + convention can keep values alive longer than is necessary. It's very + easy for both sides of the convention to end up doing extra work + because of this. + +- The caller may assert responsibility for the value. In SIL, we call + this a **guaranteed** parameter. The callee can rely on the value + staying valid for the duration of the call. + + This is optimal if the caller needs to use the value after the call + and either has responsibility for it or has a guarantee like this + for it. Therefore, this convention is particularly nice when a value + is likely to be forwarded by value a great deal. + + However, this convention does generally keep values alive longer + than is necessary, since the outermost function which passed it as + an argument will generally be forced to hold a reference for + the duration. By the same mechanism, in refcounted systems, this + convention tends to cause values to have multiple retains active at + once; for example, if a copy-on-write array is created in one + function, passed to another, stored in a mutable variable, and then + modified, the callee will see a reference count of 2 and be forced + to do a structural copy. This can occur even if the caller literally + constructed the array for the sole and immediate purpose of passing + it to the callee. + +#### Analysis + +Objective-C generally uses the unowned convention for object-pointer +parameters. It is possible to mark a parameter as being consumed, which +is basically the owned convention. As a special case, in ARC we assume +that callers are responsible for keeping self values alive (including in +blocks), which is effectively the guaranteed convention. + +unowned causes a lot of problems without really solving any, in my +experience looking at ARC-generated code and optimizer output. A human +can take advantage of it, but the compiler is so frequently blocked. +There are many common idioms (like chains of functions that just add +default arguments at each step) have really awful performance because +the compiler is adding retains and releases at every single level. It's +just not a good convention to adopt by default. However, we might want +to consider allowing specific function parameters to opt into it; sort +comparators are a particularly interesting candidate for this. unowned +is very similar to C++'s const & for things like that. + +guaranteed is good for some things, but it causes a lot of silly code +bloat when values are really only used in one place, which is quite +common. The liveness / refcounting issues are also pretty problematic. +But there is one example that's very nice for \`guaranteed\`: self. It's +quite common for clients of a type to call multiple methods on a single +value, or for methods to dispatch to multiple other methods, which are +exactly the situations where guaranteed excels. And it's relatively +uncommon (but not unimaginable) for a non-mutating method on a +copy-on-write struct to suddenly store self aside and start mutating +that copy. + +owned is a good default for other parameters. It has some minor +performance disadvantages (unnecessary retains if you have an +unoptimizable call in a loop) and some minor code size benefits (in +common straight-line code), but frankly, both of those points pale in +importance to the ability to transfer copy-on-write structures around +without spuriously increasing reference counts. It doesn't take too many +unnecessary structural copies before any amount of reference-counting +traffic (especially the Swift-native reference-counting used in +copy-on-write structures) is basically irrelevant in comparison. + +### Result values + +There's no major semantic split in result conventions like that between +pass-by-reference and pass-by-value. In most languages, a function has +to return a value (or nothing). There are languages like C++ where +functions can return references, but that's inherently limited, because +the reference has to refer to something that exists outside the +function. If Swift ever adds a similar language mechanism, it'll have to +be memory-safe and extremely opaque, and it'll be easy to just think of +that as a kind of weird value result. So we'll just consider value +results here. + +Value results raise some of the same ownership-transfer questions as +value arguments. There's one major limitation: just like a by-reference +result, an actual unowned convention is inherently limited, because +something else other than the result value must be keeping it valid. So +that's off the table for Swift. + +What Objective-C does is something more dynamic. Most APIs in +Objective-C give you a very ephemeral guarantee about the validity of +the result: it's valid now, but you shouldn't count on it being valid +indefinitely later. This might be because the result is actually owned +by some other object somewhere, or it might be because the result has +been placed in the autorelease pool, a thread-local data structure which +will (when explicitly drained by something up the call chain) eventually +release that's been put into it. This autorelease pool can be a major +source of spurious memory growth, and in classic manual +reference-counting it was important to drain it fairly frequently. ARC's +response to this convention was to add an optimization which attempts to +prevent things from ending up in the autorelease pool; the net effect of +this optimization is that ARC ends up with an owned reference regardless +of whether the value was autoreleased. So in effect, from ARC's +perspective, these APIs still return an owned reference, mediated +through some extra runtime calls to undo the damage of the convention. + +So there's really no compelling alternative to an owned return +convention as the default in Swift. + +Physical conventions +-------------------- + +The lowest abstraction level for a calling convention is the actual +"physical" rules for the call: + +- where the caller should place argument values in registers and + memory before the call, +- how the callee should pass back the return values in registers + and/or memory after the call, and +- what invariants hold about registers and memory over the call. + +In theory, all of these could be changed in the Swift ABI. In practice, +it's best to avoid changes to the invariant rules, because those rules +could complicate Swift-to-C interoperation: + +- Assuming a higher stack alignment would require dynamic realignment + whenever Swift code is called from C. +- Assuming a different set of callee-saved registers would require + additional saves and restores when either Swift code calls C or is + called from C, depending on the exact change. That would then + inhibit some kinds of tail call. + +So we will limit ourselves to considering the rules for allocating +parameters and results to registers. Our platform C ABIs are usually +quite good at this, and it's fair to ask why Swift shouldn't just use +C's rules. There are three general answers: + +- Platform C ABIs are specified in terms of the C type system, and the + Swift type system allows things to be expressed which don't have + direct analogues in C (for example, enums with payloads). +- The layout of structures in Swift does not necessarily match their + layout in C, which means that the C rules don't necessarily cover + all the cases in Swift. +- Swift places a larger emphasis on first-class structs than C does. C + ABIs often fail to allocate even small structs to registers, or use + inefficient registers for them, and we would like to be somewhat + more aggressive than that. + +Accordingly, the Swift ABI is defined largely in terms of lowering: a +Swift function signature is translated to a C function signature with +all the aggregate arguments and results eliminated (possibly by deciding +to pass them indirectly). This lowering will be described in detail in +the final section of this whitepaper. + +However, there are some specific circumstances where we'd like to +deviate from the platform ABI: + +### Aggregate results + +As mentioned above, Swift puts a lot of focus on first-class value +types. As part of this, it's very valuable to be able to return common +value types fully in registers instead of indirectly. The magic number +here is three: it's very common for copy-on-write value types to want +about three pointers' worth of data, because that's just enough for some +sort of owner pointer plus a begin/end pair. + +Unfortunately, many common C ABIs fall slightly short of that. Even +those ABIs that do allow small structs to be returned in registers tend +to only allow two pointers' worth. So in general, Swift would benefit +from a very slightly-tweaked calling convention that allocates one or +two more registers to the result. + +### Implicit parameters + +There are several language features in Swift which require implicit +parameters: + +#### Closures + +Swift's function types are "thick" by default, meaning that a function +value carries an optional context object which is implicitly passed to +the function when it is called. This context object is +reference-counted, and it should be passed guaranteed for +straightforward reasons: + +- It's not uncommon for closures to be called many times, in which + case an owned convention would be unnecessarily expensive. +- While it's easy to imagine a closure which would want to take + responsibility for its captured values, giving it responsibility for + a retain of the context object doesn't generally allow that. The + closure would only be able to take ownership of the captured values + if it had responsibility for a *unique* reference to the context. So + the closure would have to be written to do different things based on + the uniqueness of the reference, and it would have to be able to + tear down and deallocate the context object after stealing values + from it. The optimization just isn't worth it. +- It's usually straightforward for the caller to guarantee the + validity of the context reference; worst case, a single extra + Swift-native retain/release is pretty cheap. Meanwhile, not having + that guarantee would force many closure functions to retain their + contexts, since many closures do multiple things with values from + the context object. So unowned would not be a good convention. + +Many functions don't actually need a context, however; they are +naturally "thin". It would be best if it were possible to construct a +thick function directly from a thin function without having to introduce +a thunk just to move parameters around the missing context parameter. In +the worst case, a thunk would actually require the allocation of a +context object just to store the original function pointer; but that's +only necessary when converting from a completely opaque function value. +When the source function is known statically, which is far more likely, +the thunk can just be a global function which immediately calls the +target with the correctly shuffled arguments. Still, it'd be better to +be able to avoid creating such thunks entirely. + +In order to reliably avoid creating thunks, it must be possible for code +invoking an opaque thick function to pass the context pointer in a way +that can be safely and implicitly ignored if the function happens to +actually be thin. There are two ways to achieve this: + +- The context can be passed as the final parameter. In most C calling + conventions, extra arguments can be safely ignored; this is because + most C calling conventions support variadic arguments, and such + conventions inherently can't rely on the callee knowing the extent + of the arguments. + + However, this is sub-optimal because the context is often used + repeatedly in a closure, especially at the beginning, and putting it + at the end of the argument list makes it more likely to be passed on + the stack. + +- The context can be passed in a register outside of the normal + argument sequence. Some ABIs actually even reserve a register for + this purpose; for example, on x86-64 it's %r10. Neither of the ARM + ABIs do, however. + +Having an out-of-band register would be the best solution. + +(Surprisingly, the ownership transfer convention for the context doesn't +actually matter here. You might think that an owned convention would be +prohibited, since the callee would fail to release the context and would +therefore leak it. However, a thin function should always have a nil +context, so this would be harmless.) + +Either solution works acceptably with curried partial application, since +the inner parameters can be left in place while transforming the context +into the outer parameters. However, an owned convention would either +prevent the uncurrying forwarder from tail-calling the main function or +force all the arguments to be spilled. Neither is really acceptable; one +more argument against an owned convention. (This is another example +where guaranteed works quite nicely, since the guarantees are +straightforward to extend to the main function.) + +#### self + +Methods (both static and instance) require a self parameter. In all of +these cases, it's reasonable to expect that self will used frequently, +so it's best to pass it in a register. Also, many methods call other +methods on the same object, so it's also best if the register storing +self is stable across different method signatures. + +In static methods on value types, self doesn't require any dynamic +information: there's only one value of the metatype, and there's usually +no point in passing it. + +In static methods on class types, self is a reference to the class +metadata, a single pointer. This is necessary because it could actually +be the class object of a subclass. + +In instance methods on class types, self is a reference to the instance, +again a single pointer. + +In mutating instance methods on value types, self is the address of an +object. + +In non-mutating instance methods on value types, self is a value; it may +require multiple registers, or none, or it may need to be passed +indirectly. + +All of these cases except mutating instance methods on value types can +be partially applied to create a function closure whose type is the +formal type of the method. That is, if class A has a method declared +func foo(x: Int) -> Double, then A.foo yields a function of type +(Int) -> Double. Assuming that we continue to feel that this is a +useful language feature, it's worth considered how we could support it +efficiently. The expenses associated with a partial application are (1) +the allocation of a context object and (2) needing to introduce a thunk +to forward to the original function. All else aside, we can avoid the +allocation if the representation of self is compatible with the +representation of a context object reference; this is essentially true +only if self is a class instance using Swift reference counting. +Avoiding the thunk is possible only if we successfully avoided the +allocation (since otherwise a thunk is required in order to extract the +correct self value from the allocated context object) and self is passed +in exactly the same manner as a closure context would be. + +It's unclear whether making this more efficient would really be +worthwhile on its own, but if we do support an out-of-band context +parameter, taking advantage of it for methods is essentially trivial. + +### Error handling + +The calling convention implications of Swift's error handling design +aren't yet settled. It may involve extra parameters; it may involve +extra return values. Considerations: + +- Callers will generally need to immediately check for an error. Being + able to quickly check a register would be extremely convenient. +- If the error is returned as a component of the result value, it + shouldn't be physically combined with the normal result. If the + normal result is returned in registers, it would be unfortunate to + have to do complicated logic to test for error. If the normal result + is returned indirectly, contorting the indirect result with the + error would likely prevent the caller from evaluating the + call in-place. +- It would be very convenient to be able to trivially turn a function + which can't produce an error into a function which can. This is an + operation that we expect higher-order code to have do frequently, if + it isn't completely inlined away. For example: + + // foo() expects its argument to follow the conventions of a + // function that's capable of throwing. + func foo(fn: () throws -> ()) throwsIf(fn) + + // Here we're passing foo() a function that can't throw; this is + // allowed by the subtyping rules of the language. We'd like to be + // able to do this without having to introduce a thunk that maps + // between the conventions. + func bar(fn: () -> ()) { + foo(fn) + } + +We'll consider two ways to satisfy this. + +The first is to pass a pointer argument that doesn't interfere with the +normal argument sequence. The caller would initialize the memory to a +zero value. If the callee is a throwing function, it would be expected +to write the error value into this argument; otherwise, it would +naturally ignore it. Of course, the caller then has to load from memory +to see whether there's an error. This would also either consume yet +another register not in the normal argument sequence or have to be +placed at the end of the argument list, making it more likely to be +passed on the stack. + +The second is basically the same idea, but using a register that's +otherwise callee-save. The caller would initialize the register to a +zero value. A throwing function would write the error into it; a +non-throwing function would consider it callee-save and naturally +preserve it. It would then be extremely easy to check it for an error. +Of course, this would take away a callee-save register in the caller +when calling throwing functions. Also, if the caller itself isn't +throwing, it would have to save and restore that register. + +Both solutions would allow tail calls, and the zero store could be +eliminated for direct calls to known functions that can throw. The +second is the clearly superior solution, but definitely requires more +work in the backend. + +### Default argument generators + +By default, Swift is resilient about default arguments and treats them +as essentially one part of the implementation of the function. This +means that, in general, a caller using a default argument must call a +function to emit the argument, instead of simply inlining that emission +directly into the call. + +These default argument generation functions are unlike any other because +they have very precise information about how their result will be used: +it will be placed into a specific position in specific argument list. +The only reason the caller would ever want to do anything else with the +result is if it needs to spill the value before emitting the call. + +Therefore, in principle, it would be really nice if it were possible to +tell these functions to return in a very specific way, e.g. to return +two values in the second and third argument registers, or to return a +value at a specific location relative to the stack pointer (although +this might be excessively constraining; it would be reasonable to simply +opt into an indirect return instead). The function should also preserve +earlier argument registers (although this could be tricky if the default +argument generator is in a generic context and therefore needs to be +passed type-argument information). + +This enhancement is very easy to postpone because it doesn't affect any +basic language mechanics. The generators are always called directly, and +they're inherently attached to a declaration, so it's quite easy to take +any particular generator and compatibly enhance it with a better +convention. + +### ARM32 + +Most of the platforms we support have pretty good C calling conventions. +The exceptions are i386 (for the iOS simulator) and ARM32 (for iOS). We +really, really don't care about i386, but iOS on ARM32 is still an +important platform. Switching to a better physical calling convention +(only for calls from Swift to Swift, of course) would be a major +improvement. + +It would be great if this were as simple as flipping a switch, but +unfortunately the obvious convention to switch to (AAPCS-VFP) has a +slightly different set of callee-save registers: iOS treats r9 as a +scratch register. So we'd really want a variant of AAPCS-VFP that did +the same. We'd also need to make sure that SJ/LJ exceptions weren't +disturbed by this calling convention; we aren't really *supporting* +exception propagation through Swift frames, but completely breaking +propagation would be unfortunate, and we may need to be able to *catch* +exceptions. + +So this would also require some amount of additional support from the +backend. + +Function signature lowering +--------------------------- + +Function signatures in Swift are lowered in two phases. + +### Semantic lowering + +The first phase is a high-level semantic lowering, which does a number +of things: + +- It determines a high-level calling convention: specifically, whether + the function must match the C calling convention or the Swift + calling convention. +- It decides the types of the parameters: + - Functions exported for the purposes of C or Objective-C may need + to use bridged types rather than Swift's native types. For + example, a function that formally returns Swift's String type + may be bridged to return an NSString reference instead. + - Functions which are values, not simply immediately called, may + need their types lowered to follow to match a specific generic + abstraction pattern. This applies to functions that are + parameters or results of the outer function signature. +- It identifies specific arguments and results which *must* be passed + indirectly: + - Some types are inherently address-only: + - The address of a weak reference must be registered with the + runtime at all times; therefore, any struct with a weak + field must always be passed indirectly. + - An existential type (if not class-bounded) may contain an + inherently address-only value, or its layout may be + sensitive to its current address. + - A value type containing an inherently address-only type as a + field or case payload becomes itself + inherently address-only. + - Some types must be treated as address-only because their layout + is not known statically: + - The layout of a resilient value type may change in a later + release; the type may even become inherently address-only by + adding a weak reference. + - In a generic context, the layout of a type may be dependent + on a type parameter. The type parameter might even be + inherently address-only at runtime. + - A value type containing a type whose layout isn't known + statically itself generally will not have a layout that can + be known statically. + - Other types must be passed or returned indirectly because the + function type uses an abstraction pattern that requires it. For + example, a generic map function expects a function that takes a + T and returns a U; the generic implementation of map will expect + these values to be passed indirectly because their layout isn't + statically known. Therefore, the signature of a function + intended to be passed as this argument must pass them + indirectly, even if they are actually known statically to be + non-address-only types like (e.g.) Int and Float. +- It expands tuples in the parameter and result types. This is done at + this level both because it is affected by abstraction patterns and + because different tuple elements may use different + ownership conventions. (This is most likely for imported APIs, where + it's the tuple elements that correspond to specific C or + Objective-C parameters.) + + This completely eliminates top-level tuple types from the function + signature except when they are a target of abstraction and thus are + passed indirectly. (A function with type (Float, Int) -> Float + can be abstracted as (T) -> U, where T == (Float, Int).) + +- It determines ownership conventions for all parameters and results. + +After this phase, a function type consists of an abstract calling +convention, a list of parameters, and a list of results. A parameter is +a type, a flag for indirectness, and an ownership convention. A result +is a type, a flag for indirectness, and an ownership convention. +(Results need ownership conventions only for non-Swift calling +conventions.) Types will not be tuples unless they are indirect. + +Semantic lowering may also need to mark certain parameters and results +as special, for the purposes of the special-case physical treatments of +self, closure contexts, and error results. + +### Physical lowering + +The second phase of lowering translates a function type produced by +semantic lowering into a C function signature. If the function involves +a parameter or result with special physical treatment, physical lowering +initially ignores this value, then adds in the special treatment as +agreed upon with the backend. + +#### General expansion algorithm + +Central to the operation of the physical-lowering algorithm is the +**generic expansion algorithm**. This algorithm turns any +non-address-only Swift type in a sequence of zero or more **legal +type**, where a legal type is either: + +- an integer type, with a power-of-two size no larger than the maximum + integer size supported by C on the target, +- a floating-point type supported by the target, or +- a vector type supported by the target. + +Obviously, this is target-specific. The target also specifies a maximum +voluntary integer size. The legal type sequence only contains vector +types or integer types larger than the maximum voluntary size when the +type was explicit in the input. + +Pointers are represented as integers in the legal type sequence. We +assume there's never a reason to differentiate them in the ABI as long +as the effect of address spaces on pointer size is taken into account. +If that's not true, this algorithm should be adjusted. + +The result of the algorithm also associates each legal type with an +offset. This information is sufficient to reconstruct an object in +memory from a series of values and vice-versa. + +The algorithm proceeds in two steps. + +##### Typed layouts + +First, the type is recursively analyzed to produce a **typed layout**. A +typed layout associates ranges of bytes with either (1) a legal type +(whose storage size must match the size of the associated byte range), +(2) the special type **opaque**, or (3) the special type **empty**. +Adjacent ranges mapped to **opaque** or **empty** can be combined. + +For most of the types in Swift, this process is obvious: they either +correspond to an obvious legal type (e.g. thick metatypes are +pointer-sized integers), or to an obvious sequence of scalars (e.g. +class existentials are a sequence of pointer-sized integers). Only a few +cases remain: + +- Integer types that are not legal types should be mapped as opaque. +- Vector types that are not legal types should be broken into smaller + vectors, if their size is an even multiple of a legal vector type, + or else broken into their components. (This rule may need + some tinkering.) +- Tuples and structs are mapped by merging the typed layouts of the + fields, as padded out to the extents of the aggregate with + empty-mapped ranges. Note that, if fields do not overlap, this is + equivalent to concatenating the typed layouts of the fields, in + address order, mapping internal padding to empty. Bit-fields should + map the bits they occupy to opaque. + + For example, given the following struct type: + + struct FlaggedPair { + var flag: Bool + var pair: (MyClass, Float) + } + + If Swift performs naive, C-like layout of this structure, and this + is a 64-bit platform, typed layout is mapped as follows: + + FlaggedPair.flag := [0: i1, ] + FlaggedPair.pair := [ 8-15: i64, 16-19: float] + FlaggedPair := [0: i1, 8-15: i64, 16-19: float] + + If Swift instead allocates flag into the spare (little-endian) low + bits of pair.0, the typed layout map would be: + + FlaggedPair.flag := [0: i1 ] + FlaggedPair.pair := [0-7: i64, 8-11: float] + FlaggedPair := [0-7: opaque, 8-11: float] + +- Unions (imported from C) are mapped by merging the typed layouts of + the fields, as padded out to the extents of the aggregate with + empty-mapped ranges. This will often result in a + fully-opaque mapping. +- Enums are mapped by merging the typed layouts of the cases, as + padded out to the extents of the aggregate with empty-mapped ranges. + A case's typed layout consists of the typed layout of the case's + directly-stored payload (if any), merged with the typed layout for + its discriminator. We assume that checking for a discriminator + involves a series of comparisons of bits extracted from + non-overlapping ranges of the value; the typed layout of a + discriminator maps all these bits to opaque and the rest to empty. + + For example, given the following enum type: + + enum Sum { + case Yes(MyClass) + case No(Float) + case Maybe + } + + If Swift, in its infinite wisdom, decided to lay this out + sequentially, and to use invalid pointer values the class to + indicate that the other cases are present, the layout would look as + follows: + + Sum.Yes.payload := [0-7: i64 ] + Sum.Yes.discriminator := [0-7: opaque ] + Sum.Yes := [0-7: opaque ] + Sum.No.payload := [ 8-11: float] + Sum.No.discriminator := [0-7: opaque ] + Sum.No := [0-7: opaque, 8-11: float] + Sum.Maybe := [0-7: opaque ] + Sum := [0-7: opaque, 8-11: float] + + If Swift instead chose to just use a discriminator byte, the layout + would look as follows: + + Sum.Yes.payload := [0-7: i64 ] + Sum.Yes.discriminator := [ 8: opaque] + Sum.Yes := [0-7: i64, 8: opaque] + Sum.No.payload := [0-3: float ] + Sum.No.discriminator := [ 8: opaque] + Sum.No := [0-3: float, 8: opaque] + Sum.Maybe := [ 8: opaque] + Sum := [0-8: opaque ] + + If Swift chose to use spare low (little-endian) bits in the class + pointer, and to offset the float to make this possible, the layout + would look as follows: + + Sum.Yes.payload := [0-7: i64 ] + Sum.Yes.discriminator := [0: opaque ] + Sum.Yes := [0-7: opaque ] + Sum.No.payload := [ 4-7: float] + Sum.No.discriminator := [0: opaque ] + Sum.No := [0: opaque, 4-7: float] + Sum.Maybe := [0: opaque ] + Sum := [0-7: opaque ] + +The merge algorithm for typed layouts is as follows. Consider two typed +layouts L and R. A range from L is said to *conflict* with a range from +R if they intersect and they are mapped as different non-empty types. If +two ranges conflict, and either range is mapped to a vector, replace it +with mapped ranges for the vector elements. If two ranges conflict, and +neither range is mapped to a vector, map them both to opaque, combining +them with adjacent opaque ranges as necessary. If a range is mapped to a +non-empty type, and the bytes in the range are all mapped as empty in +the other map, add that range-mapping to the other map. L and R should +now match perfectly; this is the result of the merge. Note that this +algorithm is both associative and commutative. + +##### Forming a legal type sequence + +Once the typed layout is constructed, it can be turned into a legal type +sequence. + +Note that this transformation is sensitive to the offsets of ranges in +the complete type. It's possible that the simplifications described here +could be integrated directly into the construction of the typed layout +without changing the results, but that's not yet proven. + +In all of these examples, the maximum voluntary integer size is 4 (i32) +unless otherwise specified. + +If any range is mapped as a non-empty, non-opaque type, but its start +offset is not a multiple of its natural alignment, remap it as opaque. +For these purposes, the natural alignment of an integer type is the +minimum of its size and the maximum voluntary integer size; the natural +alignment of any other type is its C ABI type. Combine adjacent opaque +ranges. + +For example: + + [1-2: i16, 4: i8, 6-7: i16] ==> [1-2: opaque, 4: i8, 6-7: i16] + +If any range is mapped as an integer type that is not larger than the +maximum voluntary size, remap it as opaque. Combine adjacent opaque +ranges. + +For example: + + [1-2: opaque, 4: i8, 6-7: i16] ==> [1-2: opaque, 4: opaque, 6-7: opaque] + [0-3: i32, 4-11: i64, 12-13: i16] ==> [0-3: opaque, 4-11: i64, 12-13: opaque] + +An *aligned storage unit* is an N-byte-aligned range of N bytes, where N +is a power of 2 no greater than the maximum voluntary integer size. A +*maximal* aligned storage unit has a size equal to the maximum voluntary +integer size. + +Note that any remaining ranges mapped as integers must fully occupy +multiple maximal aligned storage units. + +Split all opaque ranges at the boundaries of maximal aligned storage +units. From this point on, never combine adjacent opaque ranges across +these boundaries. + +For example: + + [1-6: opaque] ==> [1-3: opaque, 4-6: opaque] + +Within each maximal aligned storage unit, find the smallest aligned +storage unit which contains all the opaque ranges. Replace the first +opaque range in the maximal aligned storage unit with a mapping from +that aligned storage unit to an integer of the aligned storage unit's +size. Remove any other opaque ranges in the maximal aligned storage +unit. Note that this can create overlapping ranges in some cases. For +this purposes of this calculation, the last maximal aligned storage unit +should be considered "full", as if the type had an infinite amount of +empty tail-padding. + +For example: + + [1-2: opaque] ==> [0-3: i32] + [0-1: opaque] ==> [0-1: i16] + [0: opaque, 2: opaque] ==> [0-3: i32] + [0-9: fp80, 10: opaque] ==> [0-9: fp80, 10: i8] + + // If maximum voluntary size is 8 (i64): + [0-9: fp80, 11: opaque, 13: opaque] ==> [0-9: fp80, 8-15: i64] + +(This assumes that fp80 is a legal type for illustrative purposes. It +would probably be a better policy for the actual x86-64 target to +consider it illegal and treat it as opaque from the start, at least when +lowering for the Swift calling convention; for C, it is important to +produce an fp80 mapping for ABI interoperation with C functions that +take or return long double by value.) + +The final legal type sequence is the sequence of types for the non-empty +ranges in the map. The associated offset for each type is the offset of +the start of the corresponding range. + +Only the final step can introduce overlapping ranges, and this is only +possible if there's a non-integer legal type which: + +- has a natural alignment less than half of the size of the maximum + voluntary integer size or +- has a store size is not a multiple of half the size of the maximum + voluntary integer size. + +On our supported platforms, these conditions are only true on x86-64, +and only of long double. + +#### Deconstruction and Reconstruction + +Given the address of an object and a legal type sequence for its type, +it's straightforward to load a valid sequence or store the sequence back +into memory. For the most part, it's sufficient to simply load or store +each value at its appropriate offset. There are two subtleties: + +- If the legal type sequence had any overlapping ranges, the integer + values should be stored first to prevent overwriting parts of the + other values they overlap. +- Care must be taken with the final values in the sequence; integer + values may extend slightly beyond the ordinary storage size of the + argument type. This is usually easy to compensate for. + +The value sequence essentially has the same semantics that the value in +memory would have: any bits that aren't part of the actual +representation of the original type have a completely unspecified value. + +#### Forming a C function signature + +As mentioned before, in principle the process of physical lowering turns +a semantically-lowered Swift function type (in implementation terms, a +SILFunctionType) into a C function signature, which can then be lowered +according to the usual rules for the ABI. This is, in fact, what we do +when trying to match a C calling convention. However, for the native +Swift calling convention, because we actively want to use more +aggressive rules for results, we instead build an LLVM function type +directly. We first construct a direct result type that we're certain the +backend knows how to interpret according to our more aggressive desired +rules, and then we use the expansion algorithm to construct a parameter +sequence consisting solely of types with obvious ABI lowering that the +backend can reliably handle. This bypasses the need to consult Clang for +our own native calling convention. + +We have this generic expansion algorithm, but it's important to +understand that the physical lowering process does not just naively use +the results of this algorithm. The expansion algorithm will happily +expand an arbitrary structure; if that structure is very large, the +algorithm might turn it into hundreds of values. It would be foolish to +pass it as an argument that way; it would use up all the argument +registers and basically turn into a very inefficient memcpy, and if the +caller wanted it all in one place, they'd have to very painstakingly +reassemble. It's much better to pass large structures indirectly. And +with result values, we really just don't have a choice; there's only so +many registers you can use before you have to give up and return +indirectly. Therefore, even in the Swift native convention, the +expansion algorithm is basically used as a first pass. A second pass +then decides whether the expanded sequence is actually reasonable to +pass directly. + +Recall that one aspect of the semantically-lowered Swift function type +is whether we should be matching the C calling convention or not. The +following algorithm here assumes that the importer and semantic lowering +have conspired in a very particular way to make that possible. +Specifically, we assume is that an imported C function type, lowered +semantically by Swift, will follow some simple structural rules: + +- If there was a by-value struct or union parameter or result in the + imported C type, it will correspond to a by-value direct parameter + or return type in Swift, and the Swift type will be a nominal type + whose declaration links back to the original C declaration. +- Any other parameter or result will be transformed by the importer + and semantic lowering to a type that the generic expansion algorithm + will expand to a single legal type whose representation is + ABI-compatible with the original parameter. For example, an imported + pointer type will eventually expand to an integer of pointer size. +- There will be at most one result in the lowered Swift type, and it + will be direct. + +Given this, we go about lowering the function type as follows. Recall +that, when matching the C calling convention, we're building a C +function type; but that when matching the Swift native calling +convention, we're building an LLVM function type directly. + +##### Results + +The first step is to consider the results of the function. + +There's a different set of rules here when we're matching the C calling +convention. If there's a single direct result type, and it's a nominal +type imported from Clang, then the result type of the C function type is +that imported Clang type. Otherwise, concatenate the legal type +sequences from the direct results. If this yields an empty sequence, the +result type is void. If it yields a single legal type, the result type +is the corresponding Clang type. No other could actually have come from +an imported C declaration, so we don't have any real compatibility +requirements; for the convenience of interoperation, this is handled by +constructing a new C struct which contains the corresponding Clang types +for the legal type sequence as its fields. + +Otherwise, we are matching the Swift calling convention. Concatenate the +legal type sequences from all the direct results. If target-specific +logic decides that this is an acceptable collection to return directly, +construct the appropriate IR result type to convince the backend to +handle it. Otherwise, use the void IR result type and return the +"direct" results indirectly by passing the address of a tuple combining +the original direct results (*not* the types from the legal type +sequence). + +Finally, any indirect results from the semantically-lowered function +type are simply added as pointer parameters. + +##### Parameters + +After all the results are collected, it's time to collect the +parameters. This is done one at the time, from left to right, adding +parameters to our physically-lowered type. + +If semantic lowering has decided that we have to pass the parameter +indirectly, we simply add a pointer to the type. This covers both +mandatory-indirect pass-by-value parameters and pass-by-reference +parameters. The latter can arise even in C and Objective-C. + +Otherwise, the rules are somewhat different if we're matching the C +calling convention. If the parameter is a nominal type imported from +Clang, then we just add the imported Clang type to the Clang function +type as a parameter. Otherwise, we derive the legal type sequence for +the parameter type. Again, we should only have compatibility +requirements if the legal type sequence has a single element, but for +the convenience of interoperation, we collect the corresponding Clang +types for all of the elements of the sequence. + +Finally, if we're matching the Swift calling convention, derive the +legal type sequence. If the result appears to be a reasonably small and +efficient set of parameters, add their corresponding IR types to the +function type we're building; otherwise, ignore the legal type sequence +and pass the address of the original type indirectly. + +Considerations for whether a legal type sequence is reasonable to pass +directly: + +- There probably ought to be a maximum size. Unless it's a single + 256-bit vector, it's hard to imagine wanting to pass more than, say, + 32 bytes of data as individual values. The callee may decide that it + needs to reconstruct the value for some reason, and the larger the + type gets, the more expensive this is. It may also be reasonable for + this cap to be lower on 32-bit targets, but that might be dealt with + better by the next restriction. +- There should also be a cap on the number of values. A 32-byte limit + might be reasonable for passing 4 doubles. It's probably not + reasonable for passing 8 pointers. That many values will exhaust all + the parameter registers for just a single value. 4 is probably a + reasonable cap here. +- There's no reason to require the data to be homogeneous. If a struct + contains three floats and a pointer, why force it to be passed in + memory? + +When all of the parameters have been processed in this manner, the +function type is complete. diff --git a/docs/CallingConvention.rst b/docs/CallingConvention.rst deleted file mode 100644 index a638240425051..0000000000000 --- a/docs/CallingConvention.rst +++ /dev/null @@ -1,1250 +0,0 @@ -:orphan: - -.. _CallingConvention: - -The Swift Calling Convention -**************************** - -.. contents:: - -This whitepaper discusses the Swift calling convention, at least as we -want it to be. - -It's a basic assumption in this paper that Swift shouldn't make an -implicit promise to exactly match the default platform calling -convention. That is, if a C or Objective-C programmer manages to derive the -address of a Swift function, we don't have to promise that an obvious -translation of the type of that function will be correctly callable -from C. For example, this wouldn't be guaranteed to work:: - - // In Swift: - func foo(x: Int, y: Double) -> MyClass { ... } - - // In Objective-C: - extern id _TF4main3fooFTSiSd_CS_7MyClass(intptr_t x, double y); - -We do sometimes need to be able to match C conventions, both to use -them and to generate implementations of them, but that level of -compatibility should be opt-in and site-specific. If Swift would -benefit from internally using a better convention than C/Objective-C uses, -and switching to that convention doesn't damage the dynamic abilities -of our target platforms (debugging, dtrace, stack traces, unwinding, -etc.), there should be nothing preventing us from doing so. (If we -did want to guarantee compatibility on this level, this paper would be -a lot shorter!) - -Function call rules in high-level languages have three major -components, each operating on a different abstraction level: - -* the high-level semantics of the call (pass-by-reference - vs. pass-by-value), - -* the ownership and validity conventions about argument and result - values ("+0" vs. "+1", etc.), and - -* the "physical" representation conventions of how values are actually - communicated between functions (in registers, on the stack, etc.). - -We'll tackle each of these in turn, then conclude with a detailed -discussion of function signature lowering. - -High-level semantic conventions -=============================== - -The major division in argument passing conventions between languages -is between pass-by-reference and pass-by-value languages. It's a -distinction that only really makes sense in languages with the concept -of an l-value, but Swift does, so it's pertinent. - -In general, the terms "pass-by-X" and "call-by-X" are used -interchangeably. It's unfortunate. We'll prefer "pass-by-X" for -consistency and to emphasize that these conventions are -argument-specific. - -Pass-by-reference ------------------ - -In pass-by-reference (also called pass-by-name or pass-by-address), if -`A` is an l-value expression, `foo(A)` is passed some sort of opaque -reference through which the original l-value can be modified. If `A` -is not an l-value, the language may prohibit this, or (if -pass-by-reference is the default convention) it may pass a temporary -variable containing the result of `A`. - -Don't confuse pass-by-reference with the concept of a *reference -type*. A reference type is a type whose value is a reference to a -different object; for example, a pointer type in C, or a class type in -Java or Swift. A variable of reference type can be passed by value -(copying the reference itself) or by reference (passing the variable -itself, allowing it to be changed to refer to a different object). -Note that references in C++ are a generalization of pass-by-reference, -not really a reference type; in C++, a variable of reference type -behaves completely unlike any other variable in the language. - -Also, don't confuse pass-by-reference with the physical convention of -passing an argument value indirectly. In pass-by-reference, what's -logically being passed is a reference to a tangible, user-accessible -object; changes to the original object will be visible in the -reference, and changes to the reference will be reflected in the -original object. In an indirect physical convention, the argument is -still logically an independent value, no longer associated with the -original object (if there was one). - -If every object in the language is stored in addressable memory, -pass-by-reference can be easily implemented by simply passing the -address of the object. If an l-value can have more structure than -just a single, independently-addressable object, more information may -be required from the caller. For example, an array argument in -FORTRAN can be a row or column vector from a matrix, and so arrays are -generally passed as both an address and a stride. C and C++ do have -unaddressable l-values because of bitfields, but they forbid passing -bitfields by reference (in C++) or taking their address (in either -language), which greatly simplifies pointer and reference types in -those languages. - -FORTRAN is the last remaining example of a language that defaults to -pass-by-reference. Early FORTRAN implementations famously passed -constants by passing the address of mutable global memory initialized -to the constant; if the callee modified its parameter (illegal under -the standard, but...), it literally changed the constant for future -uses. FORTRAN now allows procedures to explicitly take arguments by -value and explicitly declare that arguments must be l-values. - -However, many languages do allow parameters to be explicitly marked as -pass-by-reference. As mentioned for C++, sometimes only certain kinds -of l-values are allowed. - -Swift allows parameters to be marked as pass-by-reference with -`inout`. Arbitrary l-values can be passed. The Swift convention is -to always pass an address; if the parameter is not addressable, it -must be materialized into a temporary and then written back. See the -accessors proposal for more details about the high-level semantics of -`inout` arguments. - -Pass-by-value -------------- - -In pass-by-value, if `A` is an l-value expression, `foo(A)` copies the -current value there. Any modifications `foo` makes to its parameter -are made to this copy, not to the original l-value. - -Most modern languages are pass-by-value, with specific functions able -to opt in to pass-by-reference semantics. This is exactly what Swift -does. - -There's not much room for variation in the high-level semantics of -passing arguments by value; all the variation is in the ownership and -physical conventions. - -Ownership transfer conventions -============================== - -Arguments and results that require cleanup, like an Objective-C object -reference or a non-POD C++ object, raise two questions about -responsibility: who is responsible for cleaning it up, and when? - -These questions arise even when the cleanup is explicit in code. C's -`strdup` function returns newly-allocated memory which the caller is -responsible for freeing, but `strtok` does not. Objective-C has -standard naming conventions that describe which functions return -objects that the caller is responsible for releasing, and outside of -ARC these must be followed manually. Of course, conventions designed -to be implemented by programmers are often designed around the -simplicity of that implementation, rather than necessarily being more -efficient. - -Pass-by-reference arguments ---------------------------- - -Pass-by-reference arguments generally don't involve a *transfer* of -ownership. It's assumed that the caller will ensure that the referent -is valid at the time of the call, and that the callee will ensure that -the referent is still valid at the time of return. - -FORTRAN does actually allow parameters to be tagged as out-parameters, -where the caller doesn't guarantee the validity of the argument before -the call. Objective-C has something similar, where an indirect method -argument can be marked `out`; ARC takes advantage of this with -autoreleasing parameters to avoid a copy into the writeback temporary. -Neither of these are something we semantically care about supporting -in Swift. - -There is one other theoretically interesting convention question here: -the argument has to be valid before the call and after the call, but -does it have to valid during the call? Swift's answer to this is -generally "yes". Swift does have `inout` aliasing rules that allow a -certain amount of optimization, but the compiler is forbidden from -exploiting these rules in any way that could cause memory corruption -(at least in the absence of race conditions). So Swift has to ensure -that an `inout` argument is valid whenever it does something -(including calling an opaque function) that could potentially access -the original l-value. - -If Swift allowed local variables to be captured through `inout` -parameters, and therefore needed to pass an implicit owner parameter -along with an address, this owner parameter would behave like a -pass-by-value argument and could use any of the conventions listed -below. However, the optimal convention for this is obvious: it should -be `guaranteed`, since captures are very unlikely and callers are -almost always expected to use the value of an `inout` variable -afterwards. - -Pass-by-value arguments ------------------------ - -All conventions for this have performance trade-offs. - -We're only going to discuss *static* conventions, where the transfer -is picked at compile time. It's possible to have a *dynamic* -convention, where the caller passes a flag indicating whether it's -okay to directly take responsibility for the value, and the callee can -(conceptually) return a flag indicating whether it actually did take -responsibility for it. If copying is extremely expensive, that can be -worthwhile; otherwise, the code cost may overwhelm any other benefits. - -This discussion will ignore one particular impact of these conventions -on code size. If a function has many callers, conventions that -require more code in the caller are worse, all else aside. If a -single call site has many possible targets, conventions that require -more code in the callee are worse, all else aside. It's not really -reasonable to decide this in advance for unknown code; we could maybe -make rules about code calling system APIs, except that system APIs are -by definition locked down, and we can't change them. It's a -reasonable thing to consider changing with PGO, though. - -Responsibility -~~~~~~~~~~~~~~ - -A common refrain in this performance analysis will be whether a -function has responsibility for a value. A function has to get a -value from *somewhere*: - -* A caller is usually responsible for the return values it receives: - the callee generated the value and the caller is responsible for - destroying it. Any other convention has to rely on heavily - restricting what kind of value can be returned. (If you're thinking - about Objective-C autoreleased results, just accept this for now; - we'll talk about that later.) - -* A function isn't necessarily responsible for a value it loads from - memory. Ignoring race conditions, the function may be able to - immediately use the value without taking any specific action to keep - it valid. - -* A callee may or may not be responsible for a value passed as a - parameter, depending on the convention it was passed with. - -* A function might come from a source that doesn't necessarily make - the function responsible, but if the function takes an action which - invalidates the source before using the value, the function has to - take action to keep the value valid. At that point, the function - has responsibility for the value despite its original source. - - For example, a function `foo()` might load a reference `r` from a - global variable `x`, call an unknown function `bar()`, and then use - `r` in some way. If `bar()` can't possibly overwrite `x`, `foo()` - doesn't have to do anything to keep `r` alive across the call; - otherwise it does (e.g. by retaining it in a refcounted - environment). This is a situation where humans are often much - smarter than compilers. Of course, it's also a situation where - humans are sometimes insufficiently conservative. - -A function may also require responsibility for a value as part of its -operation: - -* Since a variable is always responsible for the current value it - stores, a function which stores a value into memory must first gain - responsibility for that value. - -* A callee normally transfers responsibility for its return value to - its caller; therefore it must gain responsibility for its return - value before returning it. - -* A caller may need to gain responsibility for a value before passing - it as an argument, depending on the parameter's ownership-transfer - convention. - -Known conventions -~~~~~~~~~~~~~~~~~ - -There are three static parameter conventions for ownership worth -considering here: - -* The caller may transfer responsibility for the value to the callee. - In SIL, we call this an **owned** parameter. - - This is optimal if the caller has responsibility for the value and - doesn't need it after the call. This is an extremely common - situation; for example, it comes up whenever a call result is - immediately used an argument. By giving the callee responsibility - for the value, this convention allows the callee to use the value at - a later point without taking any extra action to keep it alive. - - The flip side is that this convention requires a lot of extra work - when a single value is used multiple times in the caller. For - example, a value passed in every iteration of a loop will need to be - copied/retained/whatever each time. - -* The caller may provide the value without any responsibility on - either side. In SIL, we call this an **unowned** parameter. The - value is guaranteed to be valid at the moment of the call, and in - the absence of race conditions, that guarantee can be assumed to - continue unless the callee does something that might invalidate it. - As discussed above, humans are often much smarter than computers - about knowing when that's possible. - - This is optimal if the caller can acquire the value without - responsibility and the callee doesn't require responsibility of it. - In very simple code --- e.g., loading values from an array and - passing them to a comparator function which just reads a few fields - from each and returns --- this can be extremely efficient. - - Unfortunately, this convention is completely undermined if either - side has to do anything that forces it to take action to keep the - value alive. Also, if that happens on the caller side, the - convention can keep values alive longer than is necessary. It's - very easy for both sides of the convention to end up doing extra - work because of this. - -* The caller may assert responsibility for the value. In SIL, we call - this a **guaranteed** parameter. The callee can rely on the value - staying valid for the duration of the call. - - This is optimal if the caller needs to use the value after the call - and either has responsibility for it or has a guarantee like this - for it. Therefore, this convention is particularly nice when a - value is likely to be forwarded by value a great deal. - - However, this convention does generally keep values alive longer - than is necessary, since the outermost function which passed it as - an argument will generally be forced to hold a reference for the - duration. By the same mechanism, in refcounted systems, this - convention tends to cause values to have multiple retains active at - once; for example, if a copy-on-write array is created in one - function, passed to another, stored in a mutable variable, and then - modified, the callee will see a reference count of 2 and be forced - to do a structural copy. This can occur even if the caller - literally constructed the array for the sole and immediate purpose - of passing it to the callee. - -Analysis -~~~~~~~~ - -Objective-C generally uses the unowned convention for object-pointer -parameters. It is possible to mark a parameter as being consumed, -which is basically the owned convention. As a special case, in ARC we -assume that callers are responsible for keeping `self` values alive -(including in blocks), which is effectively the `guaranteed` -convention. - -`unowned` causes a lot of problems without really solving any, in my -experience looking at ARC-generated code and optimizer output. A -human can take advantage of it, but the compiler is so frequently -blocked. There are many common idioms (like chains of functions that -just add default arguments at each step) have really awful performance -because the compiler is adding retains and releases at every single -level. It's just not a good convention to adopt by default. However, -we might want to consider allowing specific function parameters to opt -into it; sort comparators are a particularly interesting candidate -for this. `unowned` is very similar to C++'s `const &` for things -like that. - -`guaranteed` is good for some things, but it causes a lot of silly -code bloat when values are really only used in one place, which is -quite common. The liveness / refcounting issues are also pretty -problematic. But there is one example that's very nice for -`guaranteed`: `self`. It's quite common for clients of a type to call -multiple methods on a single value, or for methods to dispatch to -multiple other methods, which are exactly the situations where -`guaranteed` excels. And it's relatively uncommon (but not -unimaginable) for a non-mutating method on a copy-on-write struct to -suddenly store `self` aside and start mutating that copy. - -`owned` is a good default for other parameters. It has some minor -performance disadvantages (unnecessary retains if you have an -unoptimizable call in a loop) and some minor code size benefits (in -common straight-line code), but frankly, both of those points pale in -importance to the ability to transfer copy-on-write structures around -without spuriously increasing reference counts. It doesn't take too -many unnecessary structural copies before any amount of -reference-counting traffic (especially the Swift-native -reference-counting used in copy-on-write structures) is basically -irrelevant in comparison. - -Result values -------------- - -There's no major semantic split in result conventions like that -between pass-by-reference and pass-by-value. In most languages, a -function has to return a value (or nothing). There are languages like -C++ where functions can return references, but that's inherently -limited, because the reference has to refer to something that exists -outside the function. If Swift ever adds a similar language -mechanism, it'll have to be memory-safe and extremely opaque, and -it'll be easy to just think of that as a kind of weird value result. -So we'll just consider value results here. - -Value results raise some of the same ownership-transfer questions as -value arguments. There's one major limitation: just like a -by-reference result, an actual `unowned` convention is inherently -limited, because something else other than the result value must be -keeping it valid. So that's off the table for Swift. - -What Objective-C does is something more dynamic. Most APIs in -Objective-C give you a very ephemeral guarantee about the validity of -the result: it's valid now, but you shouldn't count on it being valid -indefinitely later. This might be because the result is actually -owned by some other object somewhere, or it might be because the -result has been placed in the autorelease pool, a thread-local data -structure which will (when explicitly drained by something up the call -chain) eventually release that's been put into it. This autorelease -pool can be a major source of spurious memory growth, and in classic -manual reference-counting it was important to drain it fairly -frequently. ARC's response to this convention was to add an -optimization which attempts to prevent things from ending up in the -autorelease pool; the net effect of this optimization is that ARC ends -up with an owned reference regardless of whether the value was -autoreleased. So in effect, from ARC's perspective, these APIs still -return an owned reference, mediated through some extra runtime calls -to undo the damage of the convention. - -So there's really no compelling alternative to an owned return -convention as the default in Swift. - -Physical conventions -==================== - -The lowest abstraction level for a calling convention is the actual -"physical" rules for the call: - -* where the caller should place argument values in registers and - memory before the call, - -* how the callee should pass back the return values in registers - and/or memory after the call, and - -* what invariants hold about registers and memory over the call. - -In theory, all of these could be changed in the Swift ABI. In -practice, it's best to avoid changes to the invariant rules, because -those rules could complicate Swift-to-C interoperation: - -* Assuming a higher stack alignment would require dynamic realignment - whenever Swift code is called from C. - -* Assuming a different set of callee-saved registers would require - additional saves and restores when either Swift code calls C or is - called from C, depending on the exact change. That would then - inhibit some kinds of tail call. - -So we will limit ourselves to considering the rules for allocating -parameters and results to registers. Our platform C ABIs are usually -quite good at this, and it's fair to ask why Swift shouldn't just use -C's rules. There are three general answers: - -* Platform C ABIs are specified in terms of the C type system, and the - Swift type system allows things to be expressed which don't have - direct analogues in C (for example, enums with payloads). - -* The layout of structures in Swift does not necessarily match their - layout in C, which means that the C rules don't necessarily cover - all the cases in Swift. - -* Swift places a larger emphasis on first-class structs than C does. - C ABIs often fail to allocate even small structs to registers, or - use inefficient registers for them, and we would like to be somewhat - more aggressive than that. - -Accordingly, the Swift ABI is defined largely in terms of lowering: a -Swift function signature is translated to a C function signature with -all the aggregate arguments and results eliminated (possibly by -deciding to pass them indirectly). This lowering will be described in -detail in the final section of this whitepaper. - -However, there are some specific circumstances where we'd like to -deviate from the platform ABI: - -Aggregate results ------------------ - -As mentioned above, Swift puts a lot of focus on first-class value -types. As part of this, it's very valuable to be able to return -common value types fully in registers instead of indirectly. The -magic number here is three: it's very common for copy-on-write value -types to want about three pointers' worth of data, because that's just -enough for some sort of owner pointer plus a begin/end pair. - -Unfortunately, many common C ABIs fall slightly short of that. Even -those ABIs that do allow small structs to be returned in registers -tend to only allow two pointers' worth. So in general, Swift would -benefit from a very slightly-tweaked calling convention that allocates -one or two more registers to the result. - -Implicit parameters -------------------- - -There are several language features in Swift which require implicit -parameters: - -Closures -~~~~~~~~ - -Swift's function types are "thick" by default, meaning that a function -value carries an optional context object which is implicitly passed to -the function when it is called. This context object is -reference-counted, and it should be passed `guaranteed` for -straightforward reasons: - -* It's not uncommon for closures to be called many times, in which - case an `owned` convention would be unnecessarily expensive. - -* While it's easy to imagine a closure which would want to take - responsibility for its captured values, giving it responsibility for - a retain of the context object doesn't generally allow that. The - closure would only be able to take ownership of the captured values - if it had responsibility for a *unique* reference to the context. - So the closure would have to be written to do different things based - on the uniqueness of the reference, and it would have to be able to - tear down and deallocate the context object after stealing values - from it. The optimization just isn't worth it. - -* It's usually straightforward for the caller to guarantee the - validity of the context reference; worst case, a single extra - Swift-native retain/release is pretty cheap. Meanwhile, not having - that guarantee would force many closure functions to retain their - contexts, since many closures do multiple things with values from - the context object. So `unowned` would not be a good convention. - -Many functions don't actually need a context, however; they are -naturally "thin". It would be best if it were possible to construct a -thick function directly from a thin function without having to -introduce a thunk just to move parameters around the missing context -parameter. In the worst case, a thunk would actually require the -allocation of a context object just to store the original function -pointer; but that's only necessary when converting from a completely -opaque function value. When the source function is known statically, -which is far more likely, the thunk can just be a global function -which immediately calls the target with the correctly shuffled -arguments. Still, it'd be better to be able to avoid creating such -thunks entirely. - -In order to reliably avoid creating thunks, it must be possible for -code invoking an opaque thick function to pass the context pointer in -a way that can be safely and implicitly ignored if the function -happens to actually be thin. There are two ways to achieve this: - -* The context can be passed as the final parameter. In most C calling - conventions, extra arguments can be safely ignored; this is because - most C calling conventions support variadic arguments, and such - conventions inherently can't rely on the callee knowing the extent - of the arguments. - - However, this is sub-optimal because the context is often used - repeatedly in a closure, especially at the beginning, and putting it - at the end of the argument list makes it more likely to be passed on - the stack. - -* The context can be passed in a register outside of the normal - argument sequence. Some ABIs actually even reserve a register for - this purpose; for example, on x86-64 it's `%r10`. Neither of the - ARM ABIs do, however. - -Having an out-of-band register would be the best solution. - -(Surprisingly, the ownership transfer convention for the context -doesn't actually matter here. You might think that an `owned` -convention would be prohibited, since the callee would fail to release -the context and would therefore leak it. However, a thin function -should always have a `nil` context, so this would be harmless.) - -Either solution works acceptably with curried partial application, -since the inner parameters can be left in place while transforming the -context into the outer parameters. However, an `owned` convention -would either prevent the uncurrying forwarder from tail-calling the -main function or force all the arguments to be spilled. Neither is -really acceptable; one more argument against an `owned` convention. -(This is another example where `guaranteed` works quite nicely, since -the guarantees are straightforward to extend to the main function.) - -`self` -~~~~~~ - -Methods (both static and instance) require a `self` parameter. In all -of these cases, it's reasonable to expect that `self` will used -frequently, so it's best to pass it in a register. Also, many methods -call other methods on the same object, so it's also best if the -register storing `self` is stable across different method signatures. - -In static methods on value types, `self` doesn't require any dynamic -information: there's only one value of the metatype, and there's -usually no point in passing it. - -In static methods on class types, `self` is a reference to the class -metadata, a single pointer. This is necessary because it could -actually be the class object of a subclass. - -In instance methods on class types, `self` is a reference to the -instance, again a single pointer. - -In mutating instance methods on value types, `self` is the address of -an object. - -In non-mutating instance methods on value types, `self` is a value; it -may require multiple registers, or none, or it may need to be passed -indirectly. - -All of these cases except mutating instance methods on value types can -be partially applied to create a function closure whose type is the -formal type of the method. That is, if class `A` has a method -declared `func foo(x: Int) -> Double`, then `A.foo` yields a function -of type `(Int) -> Double`. Assuming that we continue to feel that -this is a useful language feature, it's worth considered how we could -support it efficiently. The expenses associated with a partial -application are (1) the allocation of a context object and (2) needing -to introduce a thunk to forward to the original function. All else -aside, we can avoid the allocation if the representation of `self` is -compatible with the representation of a context object reference; this -is essentially true only if `self` is a class instance using Swift -reference counting. Avoiding the thunk is possible only if we -successfully avoided the allocation (since otherwise a thunk is -required in order to extract the correct `self` value from the -allocated context object) and `self` is passed in exactly the same -manner as a closure context would be. - -It's unclear whether making this more efficient would really be -worthwhile on its own, but if we do support an out-of-band context -parameter, taking advantage of it for methods is essentially trivial. - -Error handling --------------- - -The calling convention implications of Swift's error handling design -aren't yet settled. It may involve extra parameters; it may involve -extra return values. Considerations: - -* Callers will generally need to immediately check for an error. - Being able to quickly check a register would be extremely - convenient. - -* If the error is returned as a component of the result value, it - shouldn't be physically combined with the normal result. If the - normal result is returned in registers, it would be unfortunate to - have to do complicated logic to test for error. If the normal - result is returned indirectly, contorting the indirect result with - the error would likely prevent the caller from evaluating the call - in-place. - -* It would be very convenient to be able to trivially turn a function - which can't produce an error into a function which can. This is an - operation that we expect higher-order code to have do frequently, if - it isn't completely inlined away. For example:: - - // foo() expects its argument to follow the conventions of a - // function that's capable of throwing. - func foo(fn: () throws -> ()) throwsIf(fn) - - // Here we're passing foo() a function that can't throw; this is - // allowed by the subtyping rules of the language. We'd like to be - // able to do this without having to introduce a thunk that maps - // between the conventions. - func bar(fn: () -> ()) { - foo(fn) - } - -We'll consider two ways to satisfy this. - -The first is to pass a pointer argument that doesn't interfere with -the normal argument sequence. The caller would initialize the memory -to a zero value. If the callee is a throwing function, it would be -expected to write the error value into this argument; otherwise, it -would naturally ignore it. Of course, the caller then has to load -from memory to see whether there's an error. This would also either -consume yet another register not in the normal argument sequence or -have to be placed at the end of the argument list, making it more -likely to be passed on the stack. - -The second is basically the same idea, but using a register that's -otherwise callee-save. The caller would initialize the register to a -zero value. A throwing function would write the error into it; a -non-throwing function would consider it callee-save and naturally -preserve it. It would then be extremely easy to check it for an -error. Of course, this would take away a callee-save register in the -caller when calling throwing functions. Also, if the caller itself -isn't throwing, it would have to save and restore that register. - -Both solutions would allow tail calls, and the zero store could be -eliminated for direct calls to known functions that can throw. The -second is the clearly superior solution, but definitely requires more -work in the backend. - -Default argument generators ---------------------------- - -By default, Swift is resilient about default arguments and treats them -as essentially one part of the implementation of the function. This -means that, in general, a caller using a default argument must call a -function to emit the argument, instead of simply inlining that -emission directly into the call. - -These default argument generation functions are unlike any other -because they have very precise information about how their result will -be used: it will be placed into a specific position in specific -argument list. The only reason the caller would ever want to do -anything else with the result is if it needs to spill the value before -emitting the call. - -Therefore, in principle, it would be really nice if it were possible -to tell these functions to return in a very specific way, e.g. to -return two values in the second and third argument registers, or to -return a value at a specific location relative to the stack pointer -(although this might be excessively constraining; it would be -reasonable to simply opt into an indirect return instead). The -function should also preserve earlier argument registers (although -this could be tricky if the default argument generator is in a generic -context and therefore needs to be passed type-argument information). - -This enhancement is very easy to postpone because it doesn't affect -any basic language mechanics. The generators are always called -directly, and they're inherently attached to a declaration, so it's -quite easy to take any particular generator and compatibly enhance it -with a better convention. - -ARM32 ------ - -Most of the platforms we support have pretty good C calling -conventions. The exceptions are i386 (for the iOS simulator) and -ARM32 (for iOS). We really, really don't care about i386, but iOS on -ARM32 is still an important platform. Switching to a better physical -calling convention (only for calls from Swift to Swift, of course) -would be a major improvement. - -It would be great if this were as simple as flipping a switch, but -unfortunately the obvious convention to switch to (AAPCS-VFP) has a -slightly different set of callee-save registers: iOS treats `r9` as a -scratch register. So we'd really want a variant of AAPCS-VFP that did -the same. We'd also need to make sure that SJ/LJ exceptions weren't -disturbed by this calling convention; we aren't really *supporting* -exception propagation through Swift frames, but completely breaking -propagation would be unfortunate, and we may need to be able to -*catch* exceptions. - -So this would also require some amount of additional support from the -backend. - -Function signature lowering -=========================== - -Function signatures in Swift are lowered in two phases. - -Semantic lowering ------------------ - -The first phase is a high-level semantic lowering, which does a number -of things: - -* It determines a high-level calling convention: specifically, whether - the function must match the C calling convention or the Swift - calling convention. - -* It decides the types of the parameters: - - * Functions exported for the purposes of C or Objective-C may need - to use bridged types rather than Swift's native types. For - example, a function that formally returns Swift's `String` type - may be bridged to return an `NSString` reference instead. - - * Functions which are values, not simply immediately called, may - need their types lowered to follow to match a specific generic - abstraction pattern. This applies to functions that are - parameters or results of the outer function signature. - -* It identifies specific arguments and results which *must* be passed - indirectly: - - * Some types are inherently address-only: - - * The address of a weak reference must be registered with the - runtime at all times; therefore, any `struct` with a weak field - must always be passed indirectly. - - * An existential type (if not class-bounded) may contain an - inherently address-only value, or its layout may be sensitive to - its current address. - - * A value type containing an inherently address-only type as a - field or case payload becomes itself inherently address-only. - - * Some types must be treated as address-only because their layout is - not known statically: - - * The layout of a resilient value type may change in a later - release; the type may even become inherently address-only by - adding a weak reference. - - * In a generic context, the layout of a type may be dependent on a - type parameter. The type parameter might even be inherently - address-only at runtime. - - * A value type containing a type whose layout isn't known - statically itself generally will not have a layout that can be - known statically. - - * Other types must be passed or returned indirectly because the - function type uses an abstraction pattern that requires it. For - example, a generic `map` function expects a function that takes a - `T` and returns a `U`; the generic implementation of `map` will - expect these values to be passed indirectly because their layout - isn't statically known. Therefore, the signature of a function - intended to be passed as this argument must pass them indirectly, - even if they are actually known statically to be non-address-only - types like (e.g.) `Int` and `Float`. - -* It expands tuples in the parameter and result types. This is done - at this level both because it is affected by abstraction patterns - and because different tuple elements may use different ownership - conventions. (This is most likely for imported APIs, where it's the - tuple elements that correspond to specific C or Objective-C parameters.) - - This completely eliminates top-level tuple types from the function - signature except when they are a target of abstraction and thus are - passed indirectly. (A function with type `(Float, Int) -> Float` - can be abstracted as `(T) -> U`, where `T == (Float, Int)`.) - -* It determines ownership conventions for all parameters and results. - -After this phase, a function type consists of an abstract calling -convention, a list of parameters, and a list of results. A parameter -is a type, a flag for indirectness, and an ownership convention. A -result is a type, a flag for indirectness, and an ownership -convention. (Results need ownership conventions only for non-Swift -calling conventions.) Types will not be tuples unless they are -indirect. - -Semantic lowering may also need to mark certain parameters and results -as special, for the purposes of the special-case physical treatments -of `self`, closure contexts, and error results. - -Physical lowering ------------------ - -The second phase of lowering translates a function type produced by -semantic lowering into a C function signature. If the function -involves a parameter or result with special physical treatment, -physical lowering initially ignores this value, then adds in the -special treatment as agreed upon with the backend. - -General expansion algorithm -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Central to the operation of the physical-lowering algorithm is the -**generic expansion algorithm**. This algorithm turns any -non-address-only Swift type in a sequence of zero or more **legal -type**, where a legal type is either: - -* an integer type, with a power-of-two size no larger than the maximum - integer size supported by C on the target, - -* a floating-point type supported by the target, or - -* a vector type supported by the target. - -Obviously, this is target-specific. The target also specifies a -maximum voluntary integer size. The legal type sequence only contains -vector types or integer types larger than the maximum voluntary size -when the type was explicit in the input. - -Pointers are represented as integers in the legal type sequence. We -assume there's never a reason to differentiate them in the ABI as long -as the effect of address spaces on pointer size is taken into account. -If that's not true, this algorithm should be adjusted. - -The result of the algorithm also associates each legal type with an -offset. This information is sufficient to reconstruct an object in -memory from a series of values and vice-versa. - -The algorithm proceeds in two steps. - -Typed layouts -^^^^^^^^^^^^^ - -First, the type is recursively analyzed to produce a **typed layout**. -A typed layout associates ranges of bytes with either (1) a legal type -(whose storage size must match the size of the associated byte -range), (2) the special type **opaque**, or (3) the special type -**empty**. Adjacent ranges mapped to **opaque** or **empty** can be -combined. - -For most of the types in Swift, this process is obvious: they either -correspond to an obvious legal type (e.g. thick metatypes are -pointer-sized integers), or to an obvious sequence of scalars -(e.g. class existentials are a sequence of pointer-sized integers). -Only a few cases remain: - -* Integer types that are not legal types should be mapped as opaque. - -* Vector types that are not legal types should be broken into smaller - vectors, if their size is an even multiple of a legal vector type, - or else broken into their components. (This rule may need some - tinkering.) - -* Tuples and structs are mapped by merging the typed layouts of the - fields, as padded out to the extents of the aggregate with - empty-mapped ranges. Note that, if fields do not overlap, this is - equivalent to concatenating the typed layouts of the fields, in - address order, mapping internal padding to empty. Bit-fields should - map the bits they occupy to opaque. - - For example, given the following struct type:: - - struct FlaggedPair { - var flag: Bool - var pair: (MyClass, Float) - } - - If Swift performs naive, C-like layout of this structure, and this - is a 64-bit platform, typed layout is mapped as follows:: - - FlaggedPair.flag := [0: i1, ] - FlaggedPair.pair := [ 8-15: i64, 16-19: float] - FlaggedPair := [0: i1, 8-15: i64, 16-19: float] - - If Swift instead allocates `flag` into the spare (little-endian) low - bits of `pair.0`, the typed layout map would be:: - - FlaggedPair.flag := [0: i1 ] - FlaggedPair.pair := [0-7: i64, 8-11: float] - FlaggedPair := [0-7: opaque, 8-11: float] - -* Unions (imported from C) are mapped by merging the typed layouts of - the fields, as padded out to the extents of the aggregate with - empty-mapped ranges. This will often result in a fully-opaque - mapping. - -* Enums are mapped by merging the typed layouts of the cases, as - padded out to the extents of the aggregate with empty-mapped ranges. - A case's typed layout consists of the typed layout of the case's - directly-stored payload (if any), merged with the typed layout for - its discriminator. We assume that checking for a discriminator - involves a series of comparisons of bits extracted from - non-overlapping ranges of the value; the typed layout of a - discriminator maps all these bits to opaque and the rest to empty. - - For example, given the following enum type:: - - enum Sum { - case Yes(MyClass) - case No(Float) - case Maybe - } - - If Swift, in its infinite wisdom, decided to lay this out - sequentially, and to use invalid pointer values the class to - indicate that the other cases are present, the layout would look as - follows:: - - Sum.Yes.payload := [0-7: i64 ] - Sum.Yes.discriminator := [0-7: opaque ] - Sum.Yes := [0-7: opaque ] - Sum.No.payload := [ 8-11: float] - Sum.No.discriminator := [0-7: opaque ] - Sum.No := [0-7: opaque, 8-11: float] - Sum.Maybe := [0-7: opaque ] - Sum := [0-7: opaque, 8-11: float] - - If Swift instead chose to just use a discriminator byte, the layout - would look as follows:: - - Sum.Yes.payload := [0-7: i64 ] - Sum.Yes.discriminator := [ 8: opaque] - Sum.Yes := [0-7: i64, 8: opaque] - Sum.No.payload := [0-3: float ] - Sum.No.discriminator := [ 8: opaque] - Sum.No := [0-3: float, 8: opaque] - Sum.Maybe := [ 8: opaque] - Sum := [0-8: opaque ] - - If Swift chose to use spare low (little-endian) bits in the class - pointer, and to offset the float to make this possible, the layout - would look as follows:: - - Sum.Yes.payload := [0-7: i64 ] - Sum.Yes.discriminator := [0: opaque ] - Sum.Yes := [0-7: opaque ] - Sum.No.payload := [ 4-7: float] - Sum.No.discriminator := [0: opaque ] - Sum.No := [0: opaque, 4-7: float] - Sum.Maybe := [0: opaque ] - Sum := [0-7: opaque ] - -The merge algorithm for typed layouts is as follows. Consider two -typed layouts `L` and `R`. A range from `L` is said to *conflict* -with a range from `R` if they intersect and they are mapped as -different non-empty types. If two ranges conflict, and either range -is mapped to a vector, replace it with mapped ranges for the vector -elements. If two ranges conflict, and neither range is mapped to a -vector, map them both to opaque, combining them with adjacent opaque -ranges as necessary. If a range is mapped to a non-empty type, and -the bytes in the range are all mapped as empty in the other map, add -that range-mapping to the other map. `L` and `R` should now match -perfectly; this is the result of the merge. Note that this algorithm -is both associative and commutative. - -Forming a legal type sequence -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Once the typed layout is constructed, it can be turned into a legal -type sequence. - -Note that this transformation is sensitive to the offsets of ranges in -the complete type. It's possible that the simplifications described -here could be integrated directly into the construction of the typed -layout without changing the results, but that's not yet proven. - -In all of these examples, the maximum voluntary integer size is 4 -(`i32`) unless otherwise specified. - -If any range is mapped as a non-empty, non-opaque type, but its start -offset is not a multiple of its natural alignment, remap it as opaque. -For these purposes, the natural alignment of an integer type is the -minimum of its size and the maximum voluntary integer size; the -natural alignment of any other type is its C ABI type. Combine -adjacent opaque ranges. - -For example:: - - [1-2: i16, 4: i8, 6-7: i16] ==> [1-2: opaque, 4: i8, 6-7: i16] - -If any range is mapped as an integer type that is not larger than the -maximum voluntary size, remap it as opaque. Combine adjacent opaque -ranges. - -For example:: - - [1-2: opaque, 4: i8, 6-7: i16] ==> [1-2: opaque, 4: opaque, 6-7: opaque] - [0-3: i32, 4-11: i64, 12-13: i16] ==> [0-3: opaque, 4-11: i64, 12-13: opaque] - -An *aligned storage unit* is an N-byte-aligned range of N bytes, where -N is a power of 2 no greater than the maximum voluntary integer size. -A *maximal* aligned storage unit has a size equal to the maximum -voluntary integer size. - -Note that any remaining ranges mapped as integers must fully occupy -multiple maximal aligned storage units. - -Split all opaque ranges at the boundaries of maximal aligned storage -units. From this point on, never combine adjacent opaque ranges -across these boundaries. - -For example:: - - [1-6: opaque] ==> [1-3: opaque, 4-6: opaque] - -Within each maximal aligned storage unit, find the smallest aligned -storage unit which contains all the opaque ranges. Replace the first -opaque range in the maximal aligned storage unit with a mapping from -that aligned storage unit to an integer of the aligned storage unit's -size. Remove any other opaque ranges in the maximal aligned storage -unit. Note that this can create overlapping ranges in some cases. -For this purposes of this calculation, the last maximal aligned -storage unit should be considered "full", as if the type had an -infinite amount of empty tail-padding. - -For example:: - - [1-2: opaque] ==> [0-3: i32] - [0-1: opaque] ==> [0-1: i16] - [0: opaque, 2: opaque] ==> [0-3: i32] - [0-9: fp80, 10: opaque] ==> [0-9: fp80, 10: i8] - - // If maximum voluntary size is 8 (i64): - [0-9: fp80, 11: opaque, 13: opaque] ==> [0-9: fp80, 8-15: i64] - -(This assumes that `fp80` is a legal type for illustrative purposes. -It would probably be a better policy for the actual x86-64 target to -consider it illegal and treat it as opaque from the start, at least -when lowering for the Swift calling convention; for C, it is important -to produce an `fp80` mapping for ABI interoperation with C functions -that take or return `long double` by value.) - -The final legal type sequence is the sequence of types for the -non-empty ranges in the map. The associated offset for each type is -the offset of the start of the corresponding range. - -Only the final step can introduce overlapping ranges, and this is only -possible if there's a non-integer legal type which: - -* has a natural alignment less than half of the size of the maximum - voluntary integer size or - -* has a store size is not a multiple of half the size of the maximum - voluntary integer size. - -On our supported platforms, these conditions are only true on x86-64, -and only of `long double`. - -Deconstruction and Reconstruction -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Given the address of an object and a legal type sequence for its type, -it's straightforward to load a valid sequence or store the sequence -back into memory. For the most part, it's sufficient to simply load -or store each value at its appropriate offset. There are two -subtleties: - -* If the legal type sequence had any overlapping ranges, the integer - values should be stored first to prevent overwriting parts of the - other values they overlap. - -* Care must be taken with the final values in the sequence; integer - values may extend slightly beyond the ordinary storage size of the - argument type. This is usually easy to compensate for. - -The value sequence essentially has the same semantics that the value -in memory would have: any bits that aren't part of the actual -representation of the original type have a completely unspecified -value. - -Forming a C function signature -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -As mentioned before, in principle the process of physical lowering -turns a semantically-lowered Swift function type (in implementation -terms, a SILFunctionType) into a C function signature, which can then -be lowered according to the usual rules for the ABI. This is, in -fact, what we do when trying to match a C calling convention. -However, for the native Swift calling convention, because we actively -want to use more aggressive rules for results, we instead build an -LLVM function type directly. We first construct a direct result type -that we're certain the backend knows how to interpret according to our -more aggressive desired rules, and then we use the expansion algorithm -to construct a parameter sequence consisting solely of types with -obvious ABI lowering that the backend can reliably handle. This -bypasses the need to consult Clang for our own native calling -convention. - -We have this generic expansion algorithm, but it's important to -understand that the physical lowering process does not just naively -use the results of this algorithm. The expansion algorithm will -happily expand an arbitrary structure; if that structure is very -large, the algorithm might turn it into hundreds of values. It would -be foolish to pass it as an argument that way; it would use up all the -argument registers and basically turn into a very inefficient memcpy, -and if the caller wanted it all in one place, they'd have to very -painstakingly reassemble. It's much better to pass large structures -indirectly. And with result values, we really just don't have a -choice; there's only so many registers you can use before you have to -give up and return indirectly. Therefore, even in the Swift native -convention, the expansion algorithm is basically used as a first pass. -A second pass then decides whether the expanded sequence is actually -reasonable to pass directly. - -Recall that one aspect of the semantically-lowered Swift function type -is whether we should be matching the C calling convention or not. The -following algorithm here assumes that the importer and semantic -lowering have conspired in a very particular way to make that -possible. Specifically, we assume is that an imported C function -type, lowered semantically by Swift, will follow some simple -structural rules: - -* If there was a by-value `struct` or `union` parameter or result in - the imported C type, it will correspond to a by-value direct - parameter or return type in Swift, and the Swift type will be a - nominal type whose declaration links back to the original C - declaration. - -* Any other parameter or result will be transformed by the importer - and semantic lowering to a type that the generic expansion algorithm - will expand to a single legal type whose representation is - ABI-compatible with the original parameter. For example, an - imported pointer type will eventually expand to an integer of - pointer size. - -* There will be at most one result in the lowered Swift type, and it - will be direct. - -Given this, we go about lowering the function type as follows. Recall -that, when matching the C calling convention, we're building a C -function type; but that when matching the Swift native calling -convention, we're building an LLVM function type directly. - -Results -^^^^^^^ - -The first step is to consider the results of the function. - -There's a different set of rules here when we're matching the C -calling convention. If there's a single direct result type, and it's -a nominal type imported from Clang, then the result type of the C -function type is that imported Clang type. Otherwise, concatenate the -legal type sequences from the direct results. If this yields an empty -sequence, the result type is `void`. If it yields a single legal -type, the result type is the corresponding Clang type. No other could -actually have come from an imported C declaration, so we don't have -any real compatibility requirements; for the convenience of -interoperation, this is handled by constructing a new C struct which -contains the corresponding Clang types for the legal type sequence as -its fields. - -Otherwise, we are matching the Swift calling convention. Concatenate -the legal type sequences from all the direct results. If -target-specific logic decides that this is an acceptable collection to -return directly, construct the appropriate IR result type to convince -the backend to handle it. Otherwise, use the `void` IR result type -and return the "direct" results indirectly by passing the address of a -tuple combining the original direct results (*not* the types from the -legal type sequence). - -Finally, any indirect results from the semantically-lowered function -type are simply added as pointer parameters. - -Parameters -^^^^^^^^^^ - -After all the results are collected, it's time to collect the -parameters. This is done one at the time, from left to right, adding -parameters to our physically-lowered type. - -If semantic lowering has decided that we have to pass the parameter -indirectly, we simply add a pointer to the type. This covers both -mandatory-indirect pass-by-value parameters and pass-by-reference -parameters. The latter can arise even in C and Objective-C. - -Otherwise, the rules are somewhat different if we're matching the C -calling convention. If the parameter is a nominal type imported from -Clang, then we just add the imported Clang type to the Clang function -type as a parameter. Otherwise, we derive the legal type sequence for -the parameter type. Again, we should only have compatibility -requirements if the legal type sequence has a single element, but for -the convenience of interoperation, we collect the corresponding Clang -types for all of the elements of the sequence. - -Finally, if we're matching the Swift calling convention, derive the -legal type sequence. If the result appears to be a reasonably small -and efficient set of parameters, add their corresponding IR types to -the function type we're building; otherwise, ignore the legal type -sequence and pass the address of the original type indirectly. - -Considerations for whether a legal type sequence is reasonable to pass -directly: - -* There probably ought to be a maximum size. Unless it's a single - 256-bit vector, it's hard to imagine wanting to pass more than, say, - 32 bytes of data as individual values. The callee may decide that - it needs to reconstruct the value for some reason, and the larger - the type gets, the more expensive this is. It may also be - reasonable for this cap to be lower on 32-bit targets, but that - might be dealt with better by the next restriction. - -* There should also be a cap on the number of values. A 32-byte limit - might be reasonable for passing 4 doubles. It's probably not - reasonable for passing 8 pointers. That many values will exhaust - all the parameter registers for just a single value. 4 is probably - a reasonable cap here. - -* There's no reason to require the data to be homogeneous. If a - struct contains three floats and a pointer, why force it to be - passed in memory? - -When all of the parameters have been processed in this manner, -the function type is complete. diff --git a/docs/DebuggingTheCompiler.md b/docs/DebuggingTheCompiler.md new file mode 100644 index 0000000000000..07e0bf53b00b5 --- /dev/null +++ b/docs/DebuggingTheCompiler.md @@ -0,0 +1,255 @@ +Debugging the Swift Compiler +============================ + +Abstract +-------- + +This document contains some useful information for debugging the swift +compiler and swift compiler output. + +Printing the Intermediate Representations +----------------------------------------- + +The most important thing when debugging the compiler is to examine the +IR. Here is how to dump the IR after the main phases of the swift +compiler (assuming you are compiling with optimizations enabled): + +1. **Parser**. To print the AST after parsing: + + swiftc -dump-ast -O file.swift + +2. **SILGen**. To print the SIL immediately after SILGen: + + swiftc -emit-silgen -O file.swift + +3. **Mandatory SIL passes**. To print the SIL after the mandatory + passes: + + swiftc -emit-sil -Onone file.swift + +> Well, this is not quite true, because the compiler is running some +> passes for -Onone after the mandatory passes, too. But for most +> purposes you will get what you want to see. + +1. **Performance SIL passes**. To print the SIL after the complete SIL + optimization pipeline: + + swiftc -emit-sil -O file-swift + +2. **IRGen**. To print the LLVM IR after IR generation: + + swiftc -emit-ir -Xfrontend -disable-llvm-optzns -O file.swift + +3. **LLVM passes**. To print the LLVM IR after LLVM passes: + + swiftc -emit-ir -O file.swift + +4. **Code generation**. To print the final generated code: + + swiftc -S -O file.swift + +Compilation stops at the phase where you print the output. So if you +want to print the SIL *and* the LLVM IR, you have to run the compiler +twice. The output of all these dump options (except `-dump-ast`) can be +redirected with an additional `-o ` option. + +### Debugging on SIL Level + +#### Options for Dumping the SIL + +Often it is not sufficient to dump the SIL at the beginning or end of +the optimization pipeline. The SILPassManager supports useful options to +dump the SIL also between pass runs. + +The option `-Xllvm -sil-print-all` dumps the whole SIL module after all +passes. Although it prints only functions which were changed by a pass, +the output can get *very* large. + +It is useful if you identified a problem in the final SIL and you want +to check which pass did introduce the wrong SIL. + +There are several other options available, e.g. to filter the output by +function names (`-Xllvm -sil-print-only-function`/`s`) or by pass names +(`-Xllvm -sil-print-before`/`after`/`around`). For details see +`PassManager.cpp`. + +#### Dumping the SIL and other Data in LLDB + +When debugging the swift compiler with LLDB (or Xcode, of course), there +is even a more powerful way to examine the data in the compiler, e.g. +the SIL. Following LLVM's dump() convention, many SIL classes (as well +as AST classes) provide a dump() function. You can call the dump +function with LLDB's `expression --` or `print` or `p` command. + +For example, to examine a SIL instruction: + + (lldb) p Inst->dump() + %12 = struct_extract %10 : $UnsafeMutablePointer, #UnsafeMutablePointer._rawValue // user: %13 + +To dump a whole function at the beginning of a function pass: + + (lldb) p getFunction()->dump() + +SIL modules and even functions can get very large. Often it is more +convenient to dump their contents into a file and open the file in a +separate editor. This can be done with: + + (lldb) p getFunction()->dump("myfunction.sil") + +You can also dump the CFG (control flow graph) of a function: + + (lldb) p Func->viewCFG() + +This opens a preview window containing the CFG of the function. To +continue debugging press <CTRL>-C on the LLDB prompt. Note that +this only works in Xcode if the PATH variable in the scheme's +environment setting contains the path to the dot tool. + +#### Other Utilities + +To view the CFG of a function (or code region) in a SIL file, you can +use the script `swift/utils/viewcfg`. It also works for LLVM IR files. +The script reads the SIL (or LLVM IR) code from stdin and displays the +dot graph file. Note: .dot files should be associated with the Graphviz +app. + +#### Using Breakpoints + +LLDB has very powerful breakpoints, which can be utilized in many ways +to debug the compiler and swift executables. The examples in this +section show the LLDB command lines. In Xcode you can set the breakpoint +properties by clicking 'Edit breakpoint'. + +Let's start with a simple example: sometimes you see a function in the +SIL output and you want to know where the function was created in the +compiler. In this case you can set a conditional breakpoint in +SILFunction constructor and check for the function name in the +breakpoint condition: + + (lldb) br set -c 'hasName("_TFC3nix1Xd")' -f SILFunction.cpp -l 91 + +Sometimes you want to know which optimization does insert, remove or +move a certain instruction. To find out, set a breakpoint in +`ilist_traits::addNodeToList` or +`ilist_traits::removeNodeFromList`, which are defined in +`SILInstruction.cpp`. The following command sets a breakpoint which +stops if a `strong_retain` instruction is removed: + + (lldb) br set -c 'I->getKind() == ValueKind::StrongRetainInst' -f SILInstruction.cpp -l 63 + +The condition can be made more precise e.g. by also testing in which +function this happens: + + (lldb) br set -c 'I->getKind() == ValueKind::StrongRetainInst && + I->getFunction()->hasName("_TFC3nix1Xd")' + -f SILInstruction.cpp -l 63 + +Let's assume the breakpoint hits somewhere in the middle of compiling a +large file. This is the point where the problem appears. But often you +want to break a little bit earlier, e.g. at the entrance of the +optimization's `run` function. + +To achieve this, set another breakpoint and add breakpoint commands: + + (lldb) br set -n GlobalARCOpts::run + Breakpoint 2 + (lldb) br com add 2 + > p int $n = $n + 1 + > c + > DONE + +Run the program (this can take quite a bit longer than before). When the +first breakpoint hits see what value \$n has: + + (lldb) p $n + (int) $n = 5 + +Now remove the breakpoint commands from the second breakpoint (or create +a new one) and set the ignore count to \$n minus one: + + (lldb) br delete 2 + (lldb) br set -i 4 -n GlobalARCOpts::run + +Run your program again and the breakpoint hits just before the first +breakpoint. + +Another method for accomplishing the same task is to set the ignore +count of the breakpoint to a large number, i.e.: + + (lldb) br set -i 9999999 -n GlobalARCOpts::run + +Then whenever the debugger stops next time (due to hitting another +breakpoint/crash/assert) you can list the current breakpoints: + + (lldb) br list + 1: name = 'GlobalARCOpts::run', locations = 1, resolved = 1, hit count = 85 Options: ignore: 1 enabled + +which will then show you the number of times that each breakpoint was +hit. In this case, we know that `GlobalARCOpts::run` was hit 85 times. +So, now we know to ignore swift\_getGenericMetadata 84 times, i.e.: + + (lldb) br set -i 84 -n GlobalARCOpts::run + +#### LLDB Scripts + +LLDB has powerful capabilities of scripting in python among other +languages. An often overlooked, but very useful technique is the -s +command to lldb. This essentially acts as a pseudo-stdin of commands +that lldb will read commands from. Each time lldb hits a stopping point +(i.e. a breakpoint or a crash/assert), it will run the earliest command +that has not been run yet. As an example of this consider the following +script (which without any loss of generality will be called test.lldb): + + env DYLD_INSERT_LIBRARIES=/usr/lib/libgmalloc.dylib + break set -n swift_getGenericMetadata + break mod 1 -i 83 + process launch -- --stdlib-unittest-in-process --stdlib-unittest-filter "DefaultedForwardMutableCollection>.Type.subscript(_: Range)/Set/semantics" + break set -l 224 + c + expr pattern->CreateFunction + break set -a $0 + c + dis -f + +TODO: Change this example to apply to the swift compiler instead of to +the stdlib unittests. + +Then by running `lldb test -s test.lldb`, lldb will: + +1. Enable guard malloc. +2. Set a break point on swift\_getGenericMetadata and set it to be + ignored for 83 hits. +3. Launch the application and stop at swift\_getGenericMetadata after + 83 hits have been ignored. +4. In the same file as swift\_getGenericMetadata introduce a new + breakpoint at line 224 and continue. +5. When we break at line 224 in that file, evaluate an + expression pointer. +6. Set a breakpoint at the address of the expression pointer + and continue. +7. When we hit the breakpoint set at the function pointer's address, + disassemble the function that the function pointer was passed to. + +Using LLDB scripts can enable one to use complex debugger workflows +without needing to retype the various commands perfectly every time. + +Debugging Swift Executables +--------------------------- + +One can use the previous tips for debugging the swift compiler with +swift executables as well. Here are some additional useful techniques +that one can use in Swift executables. + +### Determining the mangled name of a function in LLDB + +One problem that often comes up when debugging swift code in LLDB is +that LLDB shows the demangled name instead of the mangled name. This can +lead to mistakes where due to the length of the mangled names one will +look at the wrong function. Using the following command, one can find +the mangled name of the function in the current frame: + + (lldb) image lookup -va $pc + Address: CollectionType3[0x0000000100004db0] (CollectionType3.__TEXT.__text + 16000) + Summary: CollectionType3`ext.CollectionType3.CollectionType3.MutableCollectionType2.(subscript.materializeForSet : (Swift.Range) -> Swift.MutableSlice).(closure #1) + Module: file = "/Volumes/Files/work/solon/build/build-swift/validation-test-macosx-x86_64/stdlib/Output/CollectionType.swift.gyb.tmp/CollectionType3", arch = "x86_64" + Symbol: id = {0x0000008c}, range = [0x0000000100004db0-0x00000001000056f0), name="ext.CollectionType3.CollectionType3.MutableCollectionType2.(subscript.materializeForSet : (Swift.Range) -> Swift.MutableSlice).(closure #1)", mangled="_TFFeRq_15CollectionType322MutableCollectionType2_S_S0_m9subscriptFGVs5Rangeqq_s16MutableIndexable5Index_GVs12MutableSliceq__U_FTBpRBBRQPS0_MS4__T_" diff --git a/docs/DebuggingTheCompiler.rst b/docs/DebuggingTheCompiler.rst deleted file mode 100644 index 9faf8b9cbec13..0000000000000 --- a/docs/DebuggingTheCompiler.rst +++ /dev/null @@ -1,260 +0,0 @@ -:orphan: - -Debugging the Swift Compiler -============================ - -.. contents:: - -Abstract --------- - -This document contains some useful information for debugging the -swift compiler and swift compiler output. - -Printing the Intermediate Representations ------------------------------------------ - -The most important thing when debugging the compiler is to examine the IR. -Here is how to dump the IR after the main phases of the swift compiler -(assuming you are compiling with optimizations enabled): - -#. **Parser**. To print the AST after parsing:: - - swiftc -dump-ast -O file.swift - -#. **SILGen**. To print the SIL immediately after SILGen:: - - swiftc -emit-silgen -O file.swift - -#. **Mandatory SIL passes**. To print the SIL after the mandatory passes:: - - swiftc -emit-sil -Onone file.swift - - Well, this is not quite true, because the compiler is running some passes - for -Onone after the mandatory passes, too. But for most purposes you will - get what you want to see. - -#. **Performance SIL passes**. To print the SIL after the complete SIL - optimization pipeline:: - - swiftc -emit-sil -O file-swift - -#. **IRGen**. To print the LLVM IR after IR generation:: - - swiftc -emit-ir -Xfrontend -disable-llvm-optzns -O file.swift - -4. **LLVM passes**. To print the LLVM IR after LLVM passes:: - - swiftc -emit-ir -O file.swift - -5. **Code generation**. To print the final generated code:: - - swiftc -S -O file.swift - -Compilation stops at the phase where you print the output. So if you want to -print the SIL *and* the LLVM IR, you have to run the compiler twice. -The output of all these dump options (except ``-dump-ast``) can be redirected -with an additional ``-o `` option. - - -Debugging on SIL Level -~~~~~~~~~~~~~~~~~~~~~~ - -Options for Dumping the SIL -``````````````````````````` - -Often it is not sufficient to dump the SIL at the beginning or end of the -optimization pipeline. -The SILPassManager supports useful options to dump the SIL also between -pass runs. - -The option ``-Xllvm -sil-print-all`` dumps the whole SIL module after all -passes. Although it prints only functions which were changed by a pass, the -output can get *very* large. - -It is useful if you identified a problem in the final SIL and you want to -check which pass did introduce the wrong SIL. - -There are several other options available, e.g. to filter the output by -function names (``-Xllvm -sil-print-only-function``/``s``) or by pass names -(``-Xllvm -sil-print-before``/``after``/``around``). -For details see ``PassManager.cpp``. - -Dumping the SIL and other Data in LLDB -`````````````````````````````````````` - -When debugging the swift compiler with LLDB (or Xcode, of course), there is -even a more powerful way to examine the data in the compiler, e.g. the SIL. -Following LLVM's dump() convention, many SIL classes (as well as AST classes) -provide a dump() function. You can call the dump function with LLDB's -``expression --`` or ``print`` or ``p`` command. - -For example, to examine a SIL instruction:: - - (lldb) p Inst->dump() - %12 = struct_extract %10 : $UnsafeMutablePointer, #UnsafeMutablePointer._rawValue // user: %13 - -To dump a whole function at the beginning of a function pass:: - - (lldb) p getFunction()->dump() - -SIL modules and even functions can get very large. Often it is more convenient -to dump their contents into a file and open the file in a separate editor. -This can be done with:: - - (lldb) p getFunction()->dump("myfunction.sil") - -You can also dump the CFG (control flow graph) of a function:: - - (lldb) p Func->viewCFG() - -This opens a preview window containing the CFG of the function. To continue -debugging press -C on the LLDB prompt. -Note that this only works in Xcode if the PATH variable in the scheme's -environment setting contains the path to the dot tool. - -Other Utilities -``````````````` - -To view the CFG of a function (or code region) in a SIL file, you can use the -script ``swift/utils/viewcfg``. It also works for LLVM IR files. -The script reads the SIL (or LLVM IR) code from stdin and displays the dot -graph file. Note: .dot files should be associated with the Graphviz app. - -Using Breakpoints -````````````````` - -LLDB has very powerful breakpoints, which can be utilized in many ways to debug -the compiler and swift executables. The examples in this section show the LLDB -command lines. In Xcode you can set the breakpoint properties by clicking 'Edit -breakpoint'. - -Let's start with a simple example: sometimes you see a function in the SIL -output and you want to know where the function was created in the compiler. -In this case you can set a conditional breakpoint in SILFunction constructor -and check for the function name in the breakpoint condition:: - - (lldb) br set -c 'hasName("_TFC3nix1Xd")' -f SILFunction.cpp -l 91 - -Sometimes you want to know which optimization does insert, remove or move a -certain instruction. To find out, set a breakpoint in -``ilist_traits::addNodeToList`` or -``ilist_traits::removeNodeFromList``, which are defined in -``SILInstruction.cpp``. -The following command sets a breakpoint which stops if a ``strong_retain`` -instruction is removed:: - - (lldb) br set -c 'I->getKind() == ValueKind::StrongRetainInst' -f SILInstruction.cpp -l 63 - -The condition can be made more precise e.g. by also testing in which function -this happens:: - - (lldb) br set -c 'I->getKind() == ValueKind::StrongRetainInst && - I->getFunction()->hasName("_TFC3nix1Xd")' - -f SILInstruction.cpp -l 63 - -Let's assume the breakpoint hits somewhere in the middle of compiling a large -file. This is the point where the problem appears. But often you want to break -a little bit earlier, e.g. at the entrance of the optimization's ``run`` -function. - -To achieve this, set another breakpoint and add breakpoint commands:: - - (lldb) br set -n GlobalARCOpts::run - Breakpoint 2 - (lldb) br com add 2 - > p int $n = $n + 1 - > c - > DONE - -Run the program (this can take quite a bit longer than before). When the first -breakpoint hits see what value $n has:: - - (lldb) p $n - (int) $n = 5 - -Now remove the breakpoint commands from the second breakpoint (or create a new -one) and set the ignore count to $n minus one:: - - (lldb) br delete 2 - (lldb) br set -i 4 -n GlobalARCOpts::run - -Run your program again and the breakpoint hits just before the first breakpoint. - -Another method for accomplishing the same task is to set the ignore count of the -breakpoint to a large number, i.e.:: - - (lldb) br set -i 9999999 -n GlobalARCOpts::run - -Then whenever the debugger stops next time (due to hitting another -breakpoint/crash/assert) you can list the current breakpoints:: - - (lldb) br list - 1: name = 'GlobalARCOpts::run', locations = 1, resolved = 1, hit count = 85 Options: ignore: 1 enabled - -which will then show you the number of times that each breakpoint was hit. In -this case, we know that ``GlobalARCOpts::run`` was hit 85 times. So, now -we know to ignore swift_getGenericMetadata 84 times, i.e.:: - - (lldb) br set -i 84 -n GlobalARCOpts::run - -LLDB Scripts -```````````` - -LLDB has powerful capabilities of scripting in python among other languages. An -often overlooked, but very useful technique is the -s command to lldb. This -essentially acts as a pseudo-stdin of commands that lldb will read commands -from. Each time lldb hits a stopping point (i.e. a breakpoint or a -crash/assert), it will run the earliest command that has not been run yet. As an -example of this consider the following script (which without any loss of -generality will be called test.lldb):: - - env DYLD_INSERT_LIBRARIES=/usr/lib/libgmalloc.dylib - break set -n swift_getGenericMetadata - break mod 1 -i 83 - process launch -- --stdlib-unittest-in-process --stdlib-unittest-filter "DefaultedForwardMutableCollection>.Type.subscript(_: Range)/Set/semantics" - break set -l 224 - c - expr pattern->CreateFunction - break set -a $0 - c - dis -f - -TODO: Change this example to apply to the swift compiler instead of to the -stdlib unittests. - -Then by running ``lldb test -s test.lldb``, lldb will: - -1. Enable guard malloc. -2. Set a break point on swift_getGenericMetadata and set it to be ignored for 83 hits. -3. Launch the application and stop at swift_getGenericMetadata after 83 hits have been ignored. -4. In the same file as swift_getGenericMetadata introduce a new breakpoint at line 224 and continue. -5. When we break at line 224 in that file, evaluate an expression pointer. -6. Set a breakpoint at the address of the expression pointer and continue. -7. When we hit the breakpoint set at the function pointer's address, disassemble - the function that the function pointer was passed to. - -Using LLDB scripts can enable one to use complex debugger workflows without -needing to retype the various commands perfectly every time. - -Debugging Swift Executables ---------------------------- - -One can use the previous tips for debugging the swift compiler with swift -executables as well. Here are some additional useful techniques that one can use -in Swift executables. - -Determining the mangled name of a function in LLDB -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -One problem that often comes up when debugging swift code in LLDB is that LLDB -shows the demangled name instead of the mangled name. This can lead to mistakes -where due to the length of the mangled names one will look at the wrong -function. Using the following command, one can find the mangled name of the -function in the current frame:: - - (lldb) image lookup -va $pc - Address: CollectionType3[0x0000000100004db0] (CollectionType3.__TEXT.__text + 16000) - Summary: CollectionType3`ext.CollectionType3.CollectionType3.MutableCollectionType2.(subscript.materializeForSet : (Swift.Range) -> Swift.MutableSlice).(closure #1) - Module: file = "/Volumes/Files/work/solon/build/build-swift/validation-test-macosx-x86_64/stdlib/Output/CollectionType.swift.gyb.tmp/CollectionType3", arch = "x86_64" - Symbol: id = {0x0000008c}, range = [0x0000000100004db0-0x00000001000056f0), name="ext.CollectionType3.CollectionType3.MutableCollectionType2.(subscript.materializeForSet : (Swift.Range) -> Swift.MutableSlice).(closure #1)", mangled="_TFFeRq_15CollectionType322MutableCollectionType2_S_S0_m9subscriptFGVs5Rangeqq_s16MutableIndexable5Index_GVs12MutableSliceq__U_FTBpRBBRQPS0_MS4__T_" diff --git a/docs/Dependency Analysis.md b/docs/Dependency Analysis.md new file mode 100644 index 0000000000000..c6add70b04690 --- /dev/null +++ b/docs/Dependency Analysis.md @@ -0,0 +1,108 @@ +Dependency Analysis +=================== + +Swift's intra-module dependency analysis is based on a "provides" / +"depends" system, which is ultimately trying to prove which files do not +need to be rebuilt. In its simplest form, every file has a list of what +it "provides" and what it "depends on", and when a file is touched, +every file that "depends on" what the first file "provides" needs to be +rebuilt. + +The golden rule of dependency analysis is to be conservative. Rebuilding +a file when you don't have to is annoying. *Not* rebuilding a file when +you *do* have to is tantamount to a debug-time miscompile! + +Kinds of Dependency +------------------- + +There are four major kinds of dependency between files: + +- `top-level`: use of an unqualified name that is looked up at module + scope, and definition of a name at module scope. This includes free + functions, top-level type definitions, and global constants + and variables. +- `nominal`: use of a particular type, in any way, and declarations + that change the "shape" of the type (the original definition, and + extensions that add conformances). The type is identified by its + mangled name. +- `member`: a two-part entry that constitutes either *providing* a + member or *accessing* a specific member of a type. This has some + complications; see below. +- `dynamic-lookup`: use of a specific member accessed through + `AnyObject`, which has special `id`-like rules for member accesses, + and definitions of `@objc` members that can be accessed this way. + +The `member` dependency kind has a special entry where the member name +is empty. This is in the "provides" set of *every file that adds +non-private members* to a type, making it a superset of provided +`nominal` entries. When listed as a dependency, it means that something +in the file needs to be recompiled whenever *any* members are added to +the type in question. This is currently used for cases of inheritance: +superclasses and protocol conformances. + +The "provides" sets for each file are computed after type-checking has +completed. The "depends" sets are tracked by instrumenting lookups in +the compiler. The most common kinds of lookups (qualified name lookup, +unqualified name lookup, and protocol conformance checks) will already +properly record dependencies, but direct lookups into a type or module +will not, nor will dependencies not modeled by a query of some kind. +These latter dependencies must be handled on a case-by-case basis. + +> **note** +> +> The compiler currently does not track `nominal` dependencies separate +> from `member` dependencies. Instead, it considers every `member` +> dependency to implicitly be a `nominal` dependency, since adding a +> protocol to a type may change its members drastically. + +Cascading vs. Non-Cascading Dependencies +---------------------------------------- + +If file A depends on file B, and file B depends on file C, does file A +depend on file C? The answer is: maybe! It depends how file B is using +file C. If all uses are inside function bodies, for example, then +changing file C only requires rebuilding file B, not file A. The +terminology here is that file B has a *non-cascading* dependency on file +C. + +By contrast, if changing file C affects the interface of file B, then +the dependency is said to be *cascading,* and changing file C would +require rebuilding both file B and file A. + +The various dependency tracking throughout the compiler will look at the +context in which information is being used and attempt to determine +whether or not a particular dependency should be considered cascading. +If there's not enough context to decide, the compiler has to go with the +conservative choice and record it as cascading. + +> **note** +> +> In the current on-disk representation of dependency information, +> cascading dependencies are the default. Non-cascading dependencies are +> marked `private` by analogy with the Swift `private` keyword. + +External Dependencies +--------------------- + +External dependencies, including imported Swift module files and Clang +headers, are tracked using a special `depends-external` set. The Swift +driver interprets this set specially and decides whether or not the +cross-module dependencies have changed. + +Because every Swift file in the module at least has its imports +resolved, currently every file in the module has the same (complete) +list of external dependencies. This means if an external dependency +changes, everything in the module is rebuilt. + +Complications +------------- + +- A file's "provides" set may be different before and after it is + compiled --declarations can be both added and removed, and other + files may depend on the declarations that were added and the + declarations that were removed. This means the dependency graph has + to be updated after each file during compilation. (This is also why + reusing build products is hard.) +- If a build stops in the middle, we need to make sure the next build + takes care of anything that was scheduled to be rebuilt but didn't + get rebuilt this time. diff --git a/docs/Dependency Analysis.rst b/docs/Dependency Analysis.rst deleted file mode 100644 index dd35f732afcca..0000000000000 --- a/docs/Dependency Analysis.rst +++ /dev/null @@ -1,115 +0,0 @@ -:orphan: - -.. @raise litre.TestsAreMissing - -=================== -Dependency Analysis -=================== - -Swift's intra-module dependency analysis is based on a "provides" / "depends" -system, which is ultimately trying to prove which files do not need to be -rebuilt. In its simplest form, every file has a list of what it "provides" and -what it "depends on", and when a file is touched, every file that "depends on" -what the first file "provides" needs to be rebuilt. - -The golden rule of dependency analysis is to be conservative. Rebuilding a file -when you don't have to is annoying. *Not* rebuilding a file when you *do* have -to is tantamount to a debug-time miscompile! - - -Kinds of Dependency -=================== - -There are four major kinds of dependency between files: - -- ``top-level``: use of an unqualified name that is looked up at module scope, - and definition of a name at module scope. This includes free functions, - top-level type definitions, and global constants and variables. - -- ``nominal``: use of a particular type, in any way, and declarations that - change the "shape" of the type (the original definition, and extensions that - add conformances). The type is identified by its mangled name. - -- ``member``: a two-part entry that constitutes either *providing* a member or - *accessing* a specific member of a type. This has some complications; see - below. - -- ``dynamic-lookup``: use of a specific member accessed through ``AnyObject``, - which has special ``id``-like rules for member accesses, and definitions of - ``@objc`` members that can be accessed this way. - -The ``member`` dependency kind has a special entry where the member name is -empty. This is in the "provides" set of *every file that adds non-private -members* to a type, making it a superset of provided ``nominal`` entries. When -listed as a dependency, it means that something in the file needs to be -recompiled whenever *any* members are added to the type in question. This is -currently used for cases of inheritance: superclasses and protocol conformances. - -The "provides" sets for each file are computed after type-checking has -completed. The "depends" sets are tracked by instrumenting lookups in the -compiler. The most common kinds of lookups (qualified name lookup, unqualified -name lookup, and protocol conformance checks) will already properly record -dependencies, but direct lookups into a type or module will not, nor will -dependencies not modeled by a query of some kind. These latter dependencies -must be handled on a case-by-case basis. - -.. note:: - - The compiler currently does not track ``nominal`` dependencies separate from - ``member`` dependencies. Instead, it considers every ``member`` dependency - to implicitly be a ``nominal`` dependency, since adding a protocol to a type - may change its members drastically. - - -Cascading vs. Non-Cascading Dependencies -======================================== - -If file A depends on file B, and file B depends on file C, does file A depend -on file C? The answer is: maybe! It depends how file B is using file C. If all -uses are inside function bodies, for example, then changing file C only -requires rebuilding file B, not file A. The terminology here is that file B has -a *non-cascading* dependency on file C. - -By contrast, if changing file C affects the interface of file B, then the -dependency is said to be *cascading,* and changing file C would require -rebuilding both file B and file A. - -The various dependency tracking throughout the compiler will look at the -context in which information is being used and attempt to determine whether or -not a particular dependency should be considered cascading. If there's not -enough context to decide, the compiler has to go with the conservative choice -and record it as cascading. - -.. note:: - - In the current on-disk representation of dependency information, cascading - dependencies are the default. Non-cascading dependencies are marked - ``private`` by analogy with the Swift ``private`` keyword. - - -External Dependencies -===================== - -External dependencies, including imported Swift module files and Clang headers, -are tracked using a special ``depends-external`` set. The Swift driver -interprets this set specially and decides whether or not the cross-module -dependencies have changed. - -Because every Swift file in the module at least has its imports resolved, -currently every file in the module has the same (complete) list of external -dependencies. This means if an external dependency changes, everything in the -module is rebuilt. - - -Complications -============= - -- A file's "provides" set may be different before and after it is compiled -- - declarations can be both added and removed, and other files may depend on the - declarations that were added and the declarations that were removed. This - means the dependency graph has to be updated after each file during - compilation. (This is also why reusing build products is hard.) - -- If a build stops in the middle, we need to make sure the next build takes - care of anything that was scheduled to be rebuilt but didn't get rebuilt this - time. diff --git a/docs/DriverInternals.md b/docs/DriverInternals.md new file mode 100644 index 0000000000000..7b09f0a9ca3fe --- /dev/null +++ b/docs/DriverInternals.md @@ -0,0 +1,117 @@ +Driver Design & Internals +========================= + +Introduction +------------ + +This document serves to describe the high-level design of the Swift 2.0 +compiler driver (which includes what the driver is intended to do, and +the approach it takes to do that), as well as the internals of the +driver (which is meant to provide a brief overview of and rationale for +how the high-level design is implemented). + +The Swift driver is not intended to be GCC/Clang compatible, as it does +not need to serve as a drop-in replacement for either driver. However, +the design of the driver is inspired by Clang's design. + +Driver Stages +------------- + +The compiler driver for Swift roughly follows the same design as Clang's +compiler driver: + +1. Parse: Command-line arguments are parsed into `Arg`s. A ToolChain is + selected based on the current platform. +2. Pipeline: Based on the arguments and inputs, a tree of `Action`s + is generated. These are the high-level processing steps that need to + occur, such as "compile this file" or "link the output of all + compilation actions". +3. Bind: The ToolChain converts the `Action`s into a set of `Job`s. + These are individual commands that need to be run, such as "ld + main.o -o main". Jobs have dependencies, but are not organized into + a tree structure. +4. Execute: The `Job`s are run in a `Compilation`, which spawns off + sub-processes for each job that needs execution. The `Compilation` + is responsible for deciding which `Job`s actually need to run, based + on dependency information provided by the output of + each sub-process. The low-level management of sub-processes is + handled by a `TaskQueue`. + +### Parse: Option parsing + +The command line arguments are parsed as options and inputs into Arg +instances. Some miscellaneous validation and normalization is performed. +Most of the implementation is provided by LLVM. + +An important part of this step is selecting a ToolChain. This is the +Swift driver's view of the current platform's set of compiler tools, and +determines how it will attempt to accomplish tasks. More on this below. + +One of the optional steps here is building an *output file map.* This +allows a build system (such as Xcode) to control the location of +intermediate output files. The output file map uses a simple JSON format +mapping inputs to a map of output paths, keyed by file type. Entries +under an input of "" refer to the top-level driver process. + +> **FIXME** +> +> Certain capabilities, like incremental builds or compilation without +> linking, currently require an output file map. This should not be +> necessary. + +### Pipeline: Converting Args into Actions + +At this stage, the driver will take the input Args and input files and +establish a graph of Actions. This details the high-level tasks that +need to be performed. The graph (a DAG) tracks dependencies between +actions, but also manages ownership. + +> **FIXME** +> +> Actions currently map one-to-one to sub-process invocations. This +> means that there are actions for things that should be implementation +> details, like generating dSYM output. + +### Build: Translating Actions into Jobs using a ToolChain + +Once we have a graph of high-level Actions, we need to translate that +into actual tasks to execute. This starts by determining the output that +each Action needs to produce based on its inputs. Then we ask the +ToolChain how to perform that Action on the current platform. The +ToolChain produces a Job, which wraps up both the output information and +the actual invocation. It also remembers which Action it came from and +any Jobs it depends on. Unlike the Action graph, Jobs are owned by a +single Compilation object and stored in a flat list. + +When a Job represents a compile of a single file, it may also be used +for dependency analysis, to determine whether it is safe to not +recompile that file in the current build. This is covered by checking if +the input has been modified since the last build; if it hasn't, we only +need to recompile if something it depends on has changed. + +### Execute: Running the Jobs in a Compilation using a TaskQueue + +A Compilation's goal is to make sure every Job in its list of Jobs is +handled. If a Job needs to be run, the Compilation attempts to +*schedule* it. If the Job's dependencies have all been completed (or +determined to be skippable), it is added to the TaskQueue; otherwise it +is marked as *blocked.* + +To support Jobs compiling individual Swift files, which may or may not +need to be run, the Compilation keeps track of a DependencyGraph. (If +file A depends on file B and file B has changed, file A needs to be +recompiled.) When a Job completes successfully, the Compilation will +both re-attempt to schedule Jobs that were directly blocked on it, and +check to see if any other Jobs now need to run based on the +DependencyGraph. See the section on Dependency +Analysis for more information. + +The Compilation's TaskQueue controls the low-level aspects of managing +subprocesses. Multiple Jobs may execute simultaneously, but +communication with the parent process (the driver) is handled on a +single thread. The level of parallelism may be controlled by a compiler +flag. + +If a Job does not finish successfully, the Compilation needs to record +which jobs have failed, so that they get rebuilt next time the user +tries to build the project. diff --git a/docs/DriverInternals.rst b/docs/DriverInternals.rst deleted file mode 100644 index 041d3c09d38b2..0000000000000 --- a/docs/DriverInternals.rst +++ /dev/null @@ -1,121 +0,0 @@ -========================= -Driver Design & Internals -========================= - -.. contents:: - :local: - -Introduction -============ - -This document serves to describe the high-level design of the Swift 2.0 compiler -driver (which includes what the driver is intended to do, and the approach it -takes to do that), as well as the internals of the driver (which is meant to -provide a brief overview of and rationale for how the high-level design is -implemented). - -The Swift driver is not intended to be GCC/Clang compatible, as it does not -need to serve as a drop-in replacement for either driver. However, the design -of the driver is inspired by Clang's design. - -Driver Stages -============= - -The compiler driver for Swift roughly follows the same design as Clang's -compiler driver: - -1. Parse: Command-line arguments are parsed into ``Arg``\ s. A ToolChain is - selected based on the current platform. -2. Pipeline: Based on the arguments and inputs, a tree of ``Action``\ s is - generated. These are the high-level processing steps that need to occur, - such as "compile this file" or "link the output of all compilation actions". -3. Bind: The ToolChain converts the ``Action``\ s into a set of ``Job``\ s. - These are individual commands that need to be run, such as - "ld main.o -o main". Jobs have dependencies, but are not organized into a - tree structure. -4. Execute: The ``Job``\ s are run in a ``Compilation``, which spawns off - sub-processes for each job that needs execution. The ``Compilation`` is - responsible for deciding which ``Job``\ s actually need to run, based on - dependency information provided by the output of each sub-process. The - low-level management of sub-processes is handled by a ``TaskQueue``. - -Parse: Option parsing -^^^^^^^^^^^^^^^^^^^^^ - -The command line arguments are parsed as options and inputs into Arg instances. -Some miscellaneous validation and normalization is performed. Most of the -implementation is provided by LLVM. - -An important part of this step is selecting a ToolChain. This is the Swift -driver's view of the current platform's set of compiler tools, and determines -how it will attempt to accomplish tasks. More on this below. - -One of the optional steps here is building an *output file map.* This allows a -build system (such as Xcode) to control the location of intermediate output -files. The output file map uses a simple JSON format mapping inputs to a map of -output paths, keyed by file type. Entries under an input of "" refer to the -top-level driver process. - -.. admonition:: FIXME - - Certain capabilities, like incremental builds or compilation without - linking, currently require an output file map. This should not be necessary. - - -Pipeline: Converting Args into Actions -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -At this stage, the driver will take the input Args and input files and -establish a graph of Actions. This details the high-level tasks that need to be -performed. The graph (a DAG) tracks dependencies between actions, but also -manages ownership. - -.. admonition:: FIXME - - Actions currently map one-to-one to sub-process invocations. This means - that there are actions for things that should be implementation details, - like generating dSYM output. - - -Build: Translating Actions into Jobs using a ToolChain -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Once we have a graph of high-level Actions, we need to translate that into -actual tasks to execute. This starts by determining the output that each Action -needs to produce based on its inputs. Then we ask the ToolChain how to perform -that Action on the current platform. The ToolChain produces a Job, which wraps -up both the output information and the actual invocation. It also remembers -which Action it came from and any Jobs it depends on. Unlike the Action graph, -Jobs are owned by a single Compilation object and stored in a flat list. - -When a Job represents a compile of a single file, it may also be used for -dependency analysis, to determine whether it is safe to not recompile that file -in the current build. This is covered by checking if the input has been -modified since the last build; if it hasn't, we only need to recompile if -something it depends on has changed. - - -Execute: Running the Jobs in a Compilation using a TaskQueue -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -A Compilation's goal is to make sure every Job in its list of Jobs is handled. -If a Job needs to be run, the Compilation attempts to *schedule* it. If the -Job's dependencies have all been completed (or determined to be skippable), it -is added to the TaskQueue; otherwise it is marked as *blocked.* - -To support Jobs compiling individual Swift files, which may or may not need to -be run, the Compilation keeps track of a DependencyGraph. (If file A depends on -file B and file B has changed, file A needs to be recompiled.) When a Job -completes successfully, the Compilation will both re-attempt to schedule Jobs -that were directly blocked on it, and check to see if any other Jobs now need -to run based on the DependencyGraph. See the section on :doc:`Dependency -Analysis` for more information. - -The Compilation's TaskQueue controls the low-level aspects of managing -subprocesses. Multiple Jobs may execute simultaneously, but communication with -the parent process (the driver) is handled on a single thread. The level of -parallelism may be controlled by a compiler flag. - -If a Job does not finish successfully, the Compilation needs to record which -jobs have failed, so that they get rebuilt next time the user tries to build -the project. diff --git a/docs/DriverParseableOutput.md b/docs/DriverParseableOutput.md new file mode 100644 index 0000000000000..f714ab862f8a7 --- /dev/null +++ b/docs/DriverParseableOutput.md @@ -0,0 +1,159 @@ +Parseable Driver Output +======================= + +Introduction +------------ + +This document serves to describe the parseable output format provided by +the Swift compiler driver with the "-parseable-output" flag. This output +format is intended to be parsed by other programs; one such use case is +to allow an IDE to construct a detailed log based on the commands the +driver issued. + +Message Format +-------------- + +The parseable output provided by the Swift driver is provided as +messages encoded in JSON objects. All messages are structured like this: + + \n + { + "kind": "", + "name": "", + "": "", + ... + }\n + +This allows the driver to pass as much information as it wants about the +ongoing compilation, and programs which parse this data can decide what +to use and what to ignore. + +Message Kinds +------------- + +The driver may emit four kinds of messages: "began", "finished", +"signalled", and "skipped". + +### Began Message + +A "began" message indicates that a new task began. As with all +task-based messages, it will include the task's PID under the "pid" key. +It may specify the task's inputs as an array of paths under the "inputs" +key. It may specify the task's outputs as an array of objects under the +"outputs" key. An "outputs" object will have two fields, a "kind" +describing the type of the output, and a "path" containing the path to +the output. A "began" message will specify the command which was +executed under the "command" key. + +Example: + + { + "kind": "began", + "name": "compile", + "pid": 12345, + "inputs": [ "/src/foo.swift" ], + "outputs": [ + { + "type": "object", + "path": "/build/foo.o" + }, + { + "type": "swiftmodule", + "path": "/build/foo.swiftmodule" + }, + { + "type": "diagnostics", + "path": "/build/foo.dia" + }, + ], + "command": "swift -frontend -c -primary-file /src/foo.swift /src/bar.swift -emit-module-path /build/foo.swiftmodule -emit-diagnostics-path /build/foo.dia" + } + +### Finished Message + +A "finished" message indicates that a task finished execution. As with +all task-based messages, it will include the task's PID under the "pid" +key. It will include the exit status of the task under the "exit-status" +key. It may include the stdout/stderr of the task under the "output" +key; if this key is missing, no output was generated by the task. + +Example: + + { + "kind": "finished", + "name": "compile", + "pid": 12345, + "exit-status": 0 + // "output" key omitted because there was no stdout/stderr. + } + +### Signalled Message + +A "signalled" message indicates that a task exited abnormally due to a +signal. As with all task-based message, it will include the task's PID +under the "pid" key. It may include an error message describing the +signal under the "error-message" key. As with the "finished" message, it +may include the stdout/stderr of the task under the "output" key; if +this key is missing, no output was generated by the task. + +Example: + + { + "kind": "signalled", + "name": "compile", + "pid": 12345, + "error-message": "Segmentation fault: 11" + // "output" key omitted because there was no stdout/stderr. + } + +### Skipped Message + +A "skipped" message indicates that the driver determined a command did +not need to run during the current compilation. A "skipped" message is +equivalent to a "began" message, with the exception that it does not +include the "pid" key. + +Example: + + { + "kind": "skipped", + "name": "compile", + "inputs": [ "/src/foo.swift" ], + "outputs": [ + { + "type": "object", + "path": "/build/foo.o" + }, + { + "type": "swiftmodule", + "path": "/build/foo.swiftmodule" + }, + { + "type": "diagnostics", + "path": "/build/foo.dia" + }, + ], + "command": "swift -frontend -c -primary-file /src/foo.swift /src/bar.swift -emit-module-path /build/foo.swiftmodule -emit-diagnostics-path /build/foo.dia" + } + +Message Names +------------- + +The name of the message identifies the kind of command the message +describes. Some valid values are: + +> - compile +> - merge-module +> - link +> - generate-dsym + +A "compile" message represents a regular Swift frontend command. A +"merge-module" message represents an invocation of the Swift frontend +which is used to merge partial swiftmodule files into a complete +swiftmodule. A "link" message indicates that the driver is invoking the +linker to produce an executable or a library. A "generate-dsym" message +indicates that the driver is invoking dsymutil to generate a dSYM. + +Parsers of this format should be resilient in the event of an unknown +name, as the driver may emit messages with new names whenever it needs +to execute a new kind of command. diff --git a/docs/DriverParseableOutput.rst b/docs/DriverParseableOutput.rst deleted file mode 100644 index 12eac0f370108..0000000000000 --- a/docs/DriverParseableOutput.rst +++ /dev/null @@ -1,167 +0,0 @@ -======================= -Parseable Driver Output -======================= - -.. contents:: - :local: - -Introduction -============ - -This document serves to describe the parseable output format provided by the -Swift compiler driver with the "-parseable-output" flag. This output format is -intended to be parsed by other programs; one such use case is to allow an IDE to -construct a detailed log based on the commands the driver issued. - -Message Format -============== - -The parseable output provided by the Swift driver is provided as messages -encoded in JSON objects. All messages are structured like this:: - - \n - { - "kind": "", - "name": "", - "": "", - ... - }\n - -This allows the driver to pass as much information as it wants about the ongoing -compilation, and programs which parse this data can decide what to use and what -to ignore. - -Message Kinds -============= - -.. contents:: - :local: - -The driver may emit four kinds of messages: "began", "finished", "signalled", -and "skipped". - -Began Message -------------- - -A "began" message indicates that a new task began. As with all task-based -messages, it will include the task's PID under the "pid" key. It may specify the -task's inputs as an array of paths under the "inputs" key. It may specify the -task's outputs as an array of objects under the "outputs" key. An "outputs" -object will have two fields, a "kind" describing the type of the output, and a -"path" containing the path to the output. A "began" message will specify the -command which was executed under the "command" key. - -Example:: - - { - "kind": "began", - "name": "compile", - "pid": 12345, - "inputs": [ "/src/foo.swift" ], - "outputs": [ - { - "type": "object", - "path": "/build/foo.o" - }, - { - "type": "swiftmodule", - "path": "/build/foo.swiftmodule" - }, - { - "type": "diagnostics", - "path": "/build/foo.dia" - }, - ], - "command": "swift -frontend -c -primary-file /src/foo.swift /src/bar.swift -emit-module-path /build/foo.swiftmodule -emit-diagnostics-path /build/foo.dia" - } - -Finished Message ----------------- - -A "finished" message indicates that a task finished execution. As with all task- -based messages, it will include the task's PID under the "pid" key. It will -include the exit status of the task under the "exit-status" key. It may include -the stdout/stderr of the task under the "output" key; if this key is missing, -no output was generated by the task. - -Example:: - - { - "kind": "finished", - "name": "compile", - "pid": 12345, - "exit-status": 0 - // "output" key omitted because there was no stdout/stderr. - } - -Signalled Message ------------------ - -A "signalled" message indicates that a task exited abnormally due to a signal. -As with all task-based message, it will include the task's PID under the "pid" -key. It may include an error message describing the signal under the -"error-message" key. As with the "finished" message, it may include the -stdout/stderr of the task under the "output" key; if this key is missing, no -output was generated by the task. - -Example:: - - { - "kind": "signalled", - "name": "compile", - "pid": 12345, - "error-message": "Segmentation fault: 11" - // "output" key omitted because there was no stdout/stderr. - } - -Skipped Message ---------------- - -A "skipped" message indicates that the driver determined a command did not need to -run during the current compilation. A "skipped" message is equivalent to a "began" -message, with the exception that it does not include the "pid" key. - -Example:: - - { - "kind": "skipped", - "name": "compile", - "inputs": [ "/src/foo.swift" ], - "outputs": [ - { - "type": "object", - "path": "/build/foo.o" - }, - { - "type": "swiftmodule", - "path": "/build/foo.swiftmodule" - }, - { - "type": "diagnostics", - "path": "/build/foo.dia" - }, - ], - "command": "swift -frontend -c -primary-file /src/foo.swift /src/bar.swift -emit-module-path /build/foo.swiftmodule -emit-diagnostics-path /build/foo.dia" - } - -Message Names -============= - -The name of the message identifies the kind of command the message describes. -Some valid values are: - - - compile - - merge-module - - link - - generate-dsym - -A "compile" message represents a regular Swift frontend command. -A "merge-module" message represents an invocation of the Swift frontend which is -used to merge partial swiftmodule files into a complete swiftmodule. A "link" -message indicates that the driver is invoking the linker to produce an -executable or a library. A "generate-dsym" message indicates that the driver is -invoking dsymutil to generate a dSYM. - -Parsers of this format should be resilient in the event of an unknown name, as -the driver may emit messages with new names whenever it needs to execute a new -kind of command. \ No newline at end of file diff --git a/docs/ErrorHandling.md b/docs/ErrorHandling.md new file mode 100644 index 0000000000000..c36c5ec40c8cb --- /dev/null +++ b/docs/ErrorHandling.md @@ -0,0 +1,704 @@ +Error Handling in Swift 2.0 +=========================== + +As a tentpole feature for Swift 2.0, we are introducing a new +first-class error handling model. This feature provides standardized +syntax and language affordances for throwing, propagating, catching, and +manipulating recoverable error conditions. + +Error handling is a well-trod path, with many different approaches in +other languages, many of them problematic in various ways. We believe +that our approach provides an elegant solution, drawing on the lessons +we've learned from other languages and fixing or avoiding some of the +pitfalls. The result is expressive and concise while still feeling +explicit, safe, and familiar; and we believe it will work beautifully +with the Cocoa APIs. + +We're intentionally not using the term "exception handling", which +carries a lot of connotations from its use in other languages. Our +proposal has some similarities to the exceptions systems in those +languages, but it also has a lot of important differences. + +Kinds of Error +-------------- + +What exactly is an "error"? There are many possible error conditions, +and they don't all make sense to handle in exactly the same way, because +they arise in different circumstances and programmers have to react to +them differently. + +We can break errors down into four categories, in increasing order of +severity: + +A **simple domain error** arises from an operation that can fail in some +obvious way and which is often invoked speculatively. Parsing an integer +from a string is a really good example. The client doesn't need a +detailed description of the error and will usually want to handle the +error immediately. These errors are already well-modeled by returning an +optional value; we don't need a more complex language solution for them. + +A **recoverable error** arises from an operation which can fail in +complex ways, but whose errors can be reasonably anticipated in advance. +Examples including opening a file or reading from a network connection. +These are the kinds of errors that Apple's APIs use NSError for today, +but there are close analogues in many other APIs, such as `errno` in +POSIX. + +Ignoring this kind of error is usually a bad idea, and it can even be +dangerous (e.g. by introducing a security hole). Developers should be +strongly encouraged to write code that handles the error. It's common +for developers to want to handle errors from different operations in the +same basic way, either by reporting the error to the user or passing the +error back to their own clients. + +These errors will be the focus on this proposal. + +The final two classes of error are outside the scope of this proposal. A +**universal error** is theoretically recoverable, but by its nature the +language can't help the programmer anticipate where it will come from. A +**logic failure** arises from a programmer mistake and should not be +recoverable at all. In our system, these kinds of errors are reported +either with Objective-C/C++ exceptions or simply by logging a message +and calling `abort()`. Both kinds of error are discussed extensively in +the rationale. Having considered them carefully, we believe that we can +address them in a later release without significant harm. + +Aspects of the Design +--------------------- + +This approach proposed here is very similar to the error handling model +manually implemented in Objective-C with the `NSError` convention. +Notably, the approach preserves these advantages of this convention: + +- Whether a method produces an error (or not) is an explicit part of + its API contract. +- Methods default to *not* producing errors unless they are + explicitly marked. +- The control flow within a function is still mostly explicit: a + maintainer can tell exactly which statements can produce an error, + and a simple inspection reveals how the function reacts to + the error. +- Throwing an error provides similar performance to allocating an + error and returning it -- it isn't an expensive, table-based stack + unwinding process. +- Cocoa APIs using standard `NSError` patterns can be imported into + this world automatically. Other common patterns (e.g. `CFError`, + `errno`) can be added to the model in future versions of Swift. + +In addition, we feel that this design improves on Objective-C's error +handling approach in a number of ways: + +- It eliminates a lot of boilerplate control-flow code for + propagating errors. +- The syntax for error handling will feel familiar to people used to + exception handling in other languages. +- Defining custom error types is simple and ties in elegantly with + Swift enums. + +As to basic syntax, we decided to stick with the familiar language of +exception handling. We considered intentionally using different terms +(like `raise` / `handle`) to try to distinguish our approach from other +languages. However, by and large, error propagation in this proposal +works like it does in exception handling, and people are inevitably +going to make the connection. Given that, we couldn't find a compelling +reason to deviate from the `throw` / `catch` legacy. + +This document just contains the basic proposal and will be very light on +rationale. We considered many different languages and programming +environments as part of making this proposal, and there's an extensive +discussion of them in the separate rationale document. For example, that +document explains why we don't simply allow all functions to throw, why +we don't propagate errors using simply an `ErrorOr` return type, and +why we don't just make error propagation part of a general monad +feature. We encourage you to read that rationale if you're interested in +understanding why we made the decisions we did. + +With that out of the way, let's get to the details of the proposal. + +Typed propagation +----------------- + +Whether a function can throw is part of its type. This applies to all +functions, whether they're global functions, methods, or closures. + +By default, a function cannot throw. The compiler statically enforces +this: anything the function does which can throw must appear in a +context which handles all errors. + +A function can be declared to throw by writing `throws` on the function +declaration or type: + + func foo() -> Int { // This function is not permitted to throw. + func bar() throws -> Int { // This function is permitted to throw. + +`throws` is written before the arrow to give a sensible and consistent +grammar for function types and implicit `()` result types, e.g.: + + func baz() throws { + + // Takes a 'callback' function that can throw. + // 'fred' itself can also throw. + func fred(callback: (UInt8) throws -> ()) throws { + + // These are distinct types. + let a : () -> () -> () + let b : () throws -> () -> () + let c : () -> () throws -> () + let d : () throws -> () throws -> () + +For curried functions, `throws` only applies to the innermost function. +This function has type `(Int) -> (Int) throws -> Int`: + + func jerry(i: Int)(j: Int) throws -> Int { + +`throws` is tracked as part of the type system: a function value must +also declare whether it can throw. Functions that cannot throw are a +subtype of functions that can, so you can use a function that can't +throw anywhere you could use a function that can: + + func rachel() -> Int { return 12 } + func donna(generator: () throws -> Int) -> Int { ... } + + donna(rachel) + +The reverse is not true, since the caller would not be prepared to +handle the error. + +A call to a function which can throw within a context that is not +allowed to throw is rejected by the compiler. + +It isn't possible to overload functions solely based on whether the +functions throw. That is, this is not legal: + + func foo() { + func foo() throws { + +A throwing method cannot override a non-throwing method or satisfy a +non-throwing protocol requirement. However, a non-throwing method can +override a throwing method or satisfy a throwing protocol requirement. + +It is valuable to be able to overload higher-order functions based on +whether an argument function throws, so this is allowed: + + func foo(callback: () throws -> Bool) { + func foo(callback: () -> Bool) { + +### `rethrows` + +Functions which take a throwing function argument (including as an +autoclosure) can be marked as `rethrows`: + + extension Array { + func map(fn: ElementType throws -> U) rethrows -> [U] + } + +It is an error if a function declared `rethrows` does not include a +throwing function in at least one of its parameter clauses. + +`rethrows` is identical to `throws`, except that the function promises +to only throw if one of its argument functions throws. + +More formally, a function is *rethrowing-only* for a function *f* if: + +- it is a throwing function parameter of *f*, +- it is a non-throwing function, or +- it is implemented within *f* (i.e. it is either *f* or a function or + closure defined therein) and it does not throw except by either: + - calling a function that is rethrowing-only for *f* or + - calling a function that is `rethrows`, passing only functions + that are rethrowing-only for *f*. + +It is an error if a `rethrows` function is not rethrowing-only for +itself. + +A `rethrows` function is considered to be a throwing function. However, +a direct call to a `rethrows` function is considered to not throw if it +is fully applied and none of the function arguments can throw. For +example: + + // This call to map is considered not to throw because its + // argument function does not throw. + let absolutePaths = paths.map { "/" + $0 } + + // This call to map is considered to throw because its + // argument function does throw. + let streams = try absolutePaths.map { try InputStream(filename: $0) } + +For now, `rethrows` is a property of declared functions, not of function +values. Binding a variable (even a constant) to a function loses the +information that the function was `rethrows`, and calls to it will use +the normal rules, meaning that they will be considered to throw +regardless of whether a non-throwing function is passed. + +For the purposes of override and conformance checking, `rethrows` lies +between `throws` and non-`throws`. That is, an ordinary throwing method +cannot override a `rethrows` method, which cannot override a +non-throwing method; but an ordinary throwing method can be overridden +by a `rethrows` method, which can be overridden by a non-throwing +method. Equivalent rules apply for protocol conformance. + +Throwing an error +----------------- + +The `throw` statement begins the propagation of an error. It always take +an argument, which can be any value that conforms to the `ErrorType` +protocol (described below). + + if timeElapsed > timeThreshold { + throw HomeworkError.Overworked + } + + throw NSError(domain: "whatever", code: 42, userInfo: nil) + +As mentioned above, attempting to throw an error out of a function not +marked `throws` is a static compiler error. + +Catching errors +--------------- + +A `catch` clause includes an optional pattern that matches the error. +This pattern can use any of the standard pattern-matching tools provided +by `switch` statements in Swift, including boolean `where` conditions. +The pattern can be omitted; if so, a `where` condition is still +permitted. If the pattern is omitted, or if it does not bind a different +name to the error, the name `error` is automatically bound to the error +as if with a `let` pattern. + +The `try` keyword is used for other purposes which it seems to fit far +better (see below), so `catch` clauses are instead attached to a +generalized `do` statement: + + // Simple do statement (without a trailing while condition), + // just provides a scope for variables defined inside of it. + do { + let x = foo() + } + + // do statement with two catch clauses. + do { + ... + + } catch HomeworkError.Overworked { + // a conditionally-executed catch clause + + } catch _ { + // a catch-all clause. + } + +As with `switch` statements, Swift makes an effort to understand whether +catch clauses are exhaustive. If it can determine it is, then the +compiler considers the error to be handled. If not, the error +automatically propagates out out of scope, either to a lexically +enclosing `catch` clause or out of the containing function (which must +be marked `throws`). + +We expect to refine the `catch` syntax with usage experience. + +`ErrorType` +----------- + +The Swift standard library will provide `ErrorType`, a protocol with a +very small interface (which is not described in this proposal). The +standard pattern should be to define the conformance of an `enum` to the +type: + + enum HomeworkError : ErrorType { + case Overworked + case Impossible + case EatenByCat(Cat) + case StopStressingMeWithYourRules + } + +The `enum` provides a namespace of errors, a list of possible errors +within that namespace, and optional values to attach to each option. + +Note that this corresponds very cleanly to the `NSError` model of an +error domain, an error code, and optional user data. We expect to import +system error domains as enums that follow this approach and implement +`ErrorType`. `NSError` and `CFError` themselves will also conform to +`ErrorType`. + +The physical representation (still being nailed down) will make it +efficient to embed an `NSError` as an `ErrorType` and vice-versa. It +should be possible to turn an arbitrary Swift `enum` that conforms to +`ErrorType` into an `NSError` by using the qualified type name as the +domain key, the enumerator as the error code, and turning the payload +into user data. + +Automatic, marked, propagation of errors +---------------------------------------- + +Once an error is thrown, Swift will automatically propagate it out of +scopes (that permit it), rather than relying on the programmer to +manually check for errors and do their own control flow. This is just a +lot less boilerplate for common error handling tasks. However, doing +this naively would introduce a lot of implicit control flow, which makes +it difficult to reason about the function's behavior. This is a serious +maintenance problem and has traditionally been a considerable source of +bugs in languages that heavily use exceptions. + +Therefore, while Swift automatically propagates errors, it requires that +statements and expressions that can implicitly throw be marked with the +`try` keyword. For example: + + func readStuff() throws { + // loadFile can throw an error. If so, it propagates out of readStuff. + try loadFile("mystuff.txt") + + // This is a semantic error; the 'try' keyword is required + // to indicate that it can throw. + var y = stream.readFloat() + + // This is okay; the try covers the entire statement. + try y += stream.readFloat() + + // This try applies to readBool(). + if try stream.readBool() { + // This try applies to both of these calls. + let x = try stream.readInt() + stream.readInt() + } + + if let err = stream.getOutOfBandError() { + // Of course, the programmer doesn't have to mark explicit throws. + throw err + } + } + +Developers can choose to "scope" the `try` very tightly by writing it +within parentheses or on a specific argument or list element: + + // Ok. + let x = (try stream.readInt()) + (try stream.readInt()) + + // Semantic error: the try only covers the parenthesized expression. + let x2 = (try stream.readInt()) + stream.readInt() + + // The try applies to the first array element. Of course, the + // developer could cover the entire array by writing the try outside. + let array = [ try foo(), bar(), baz() ] + +Some developers may wish to do this to make the specific throwing calls +very clear. Other developers may be content with knowing that something +within a statement can throw. The compiler's fixit hints will guide +developers towards inserting a single `try` that covers the entire +statement. This could potentially be controlled someday by a coding +style flag passed to the compiler. + +### `try!` + +To concisely indicate that a call is known to not actually throw at +runtime, `try` can be decorated with `!`, turning the error check into a +runtime assertion that the call does not throw. + +For the purposes of checking that all errors are handled, a `try!` +expression is considered to handle any error originating from within its +operand. + +`try!` is otherwise exactly like `try`: it can appear in exactly the +same positions and doesn't affect the type of an expression. + +Manual propagation and manipulation of errors +--------------------------------------------- + +Taking control over the propagation of errors is important for some +advanced use cases (e.g. transporting an error result across threads +when synchronizing a future) and can be more convenient or natural for +specific use cases (e.g. handling a specific call differently within a +context that otherwise allows propagation). + +As such, the Swift standard library should provide a standard Rust-like +`Result` enum, along with API for working with it, e.g.: + +- A function to evaluate an error-producing closure and capture the + result as a `Result`. +- A function to unpack a `Result` by either returning its value or + propagating the error in the current context. + +This is something that composes on top of the basic model, but that has +not been designed yet and details aren't included in this proposal. + +The name `Result` is a stand-in and needs to be designed and +reviewed, as well as the basic operations on the type. + +`defer` +------- + +Swift should provide a `defer` statement that sets up an *ad hoc* +clean-up action to be run when the current scope is exited. This +replicates the functionality of a Java-style `finally`, but more cleanly +and with less nesting. + +This is an important tool for ensuring that explicitly-managed resources +are released on all paths. Examples include closing a network connection +and freeing memory that was manually allocated. It is convenient for all +kinds of error-handling, even manual propagation and simple domain +errors, but is especially nice with automatic propagation. It is also a +crucial part of our long-term vision for universal errors. + +`defer` may be followed by an arbitrary statement. The compiler should +reject a `defer` action that might terminate early, whether by throwing +or with `return`, `break`, or `continue`. + +Example: + + if exists(filename) { + let file = open(filename, O_READ) + defer close(file) + + while let line = try file.readline() { + ... + } + + // close occurs here, at the end of the formal scope. + } + +If there are multiple defer statements in a scope, they are guaranteed +to be executed in reverse order of appearance. That is: + + let file1 = open("hello.txt") + defer close(file1) + let file2 = open("world.txt") + defer close(file2) + ... + // file2 will be closed first. + +A potential extension is to provide a convenient way to mark that a +defer action should only be taken if an error is thrown. This is a +convenient shorthand for controlling the action with a flag. We will +evaluate whether adding complexity to handle this case is justified +based on real-world usage experience. + +Importing Cocoa +--------------- + +If possible, Swift's error-handling model should transparently work with +the SDK with a minimal amount of effort from framework owners. + +We believe that we can cover the vast majority of Objective-C APIs with +`NSError**` out-parameters by importing them as `throws` and removing +the error clause from their signature. That is, a method like this one +from `NSAttributedString`: + + - (NSData *)dataFromRange:(NSRange)range + documentAttributes:(NSDictionary *)dict + error:(NSError **)error; + +would be imported as: + + func dataFromRange(range: NSRange, + documentAttributes dict: NSDictionary) throws -> NSData + +There are a number of cases to consider, but we expect that most can be +automatically imported without extra annotation in the SDK, by using a +couple of simple heuristics: + +- The most common pattern is a `BOOL` result, where a false value + means an error occurred. This seems unambiguous. +- Also common is a pointer result, where a `nil` result usually means + an error occurred. This appears to be universal in Objective-C; APIs + that can return `nil` results seem to do so via out-parameters. So + it seems to be safe to make a policy decision that it's okay to + assume that a `nil` result is an error by default. + + If the pattern for a method is that a `nil` result means it produced + an error, then the result can be imported as a non-optional type. + +- A few APIs return `void`. As far as I can tell, for all of these, + the caller is expected to check for a non-`nil` error. + +For other sentinel cases, we can consider adding a new clang attribute +to indicate to the compiler what the sentinel is: + +- There are several APIs returning `NSInteger` or `NSUInteger`. At + least some of these return 0 on error, but that doesn't seem like a + reasonable general assumption. +- `AVFoundation` provides a couple methods returning + `AVKeyValueStatus`. These produce an error if the API returned + `AVKeyValueStatusFailed`, which, interestingly enough, is not the + zero value. + +The clang attribute would specify how to test the return value for an +error. For example: + + + (NSInteger)writePropertyList:(id)plist + toStream:(NSOutputStream *)stream + format:(NSPropertyListFormat)format + options:(NSPropertyListWriteOptions)opt + error:(out NSError **)error + NS_ERROR_RESULT(0); + + - (AVKeyValueStatus)statusOfValueForKey:(NSString *)key + error:(NSError **) + NS_ERROR_RESULT(AVKeyValueStatusFailed); + +We should also provide a Clang attribute which specifies that the +correct way to test for an error is to check the out-parameter. Both of +these attributes could potentially be used by the static analyzer, not +just Swift. (For example, they could try to detect an invalid error +check.) + +Cases that do not match the automatically imported patterns and that +lack an attribute would be left unmodified (i.e., they'd keep their +NSErrorPointer argument) and considered "not awesome" in the SDK +auditing tool. These will still be usable in Swift: callers will get the +NSError back like they do today, and have to throw the result manually. + +For initializers, importing an initializer as throwing takes precedence +over importing it as failable. That is, an imported initializer with a +nullable result and an error parameter would be imported as throwing. +Throwing initializers have very similar constraints to failable +initializers; in a way, it's just a new axis of failability. + +One limitation of this approach is that we need to be able to +reconstruct the selector to use when an overload of a method is +introduced. For this reason, the import is likely to be limited to +methods where the error parameter is the last one and the corresponding +selector chunk is either `error:` or the first chunk (see below). +Empirically, this seems to do the right thing for all but two sets of +APIs in the public API: + +- The `ISyncSessionDriverDelegate` category on `NSObject` declares + half-a-dozen methods like this: + + - (BOOL)sessionDriver:(ISyncSessionDriver *)sender + didRegisterClientAndReturnError:(NSError **)outError; + + Fortunately, these delegate methods were all deprecated in Lion, and + are thus unavailable in Swift. + +- `NSFileCoordinator` has half a dozen methods where the `error:` + clause is second-to-last, followed by a block argument. These + methods are not deprecated as far as I know. + +The above translation rule would import methods like this one from +`NSDocument`: + + - (NSDocument *)duplicateAndReturnError:(NSError **)outError; + +like so: + + func duplicateAndReturnError() throws -> NSDocument + +The `AndReturnError` bit is common but far from universal; consider this +method from `NSManagedObject`: + + - (BOOL)validateForDelete:(NSError **)error; + +This would be imported as: + + func validateForDelete() throws + +This is a really nice import, and it's somewhat unfortunate that we +can't import `duplicateAndReturnError:` as `duplicate()`. + +Potential future extensions to this model +----------------------------------------- + +We believe that the proposal above is sufficient to provide a huge step +forward in error handling in Swift programs, but there is always more to +consider in the future. Some specific things we've discussed (and may +come back to in the future) but don't consider to be core to the Swift +2.0 model are: + +### Higher-order polymorphism + +We should make it easy to write higher-order functions that behave +polymorphically with respect to whether their arguments throw. This can +be done in a fairly simple way: a function can declare that it throws if +any of a set of named arguments do. As an example (using strawman +syntax): + + func map(array: [T], fn: T -> U) throwsIf(fn) -> [U] { + ... + } + +There's no need for a more complex logical operator than disjunction for +normal higher-order stuff. + +This feature is highly desired (e.g. it would allow many otherwise +redundant overloads to be collapsed into a single definition), but it +may or may not make it into Swift 2.0 based on schedule limitations. + +### Generic polymorphism + +For similar reasons to higher-order polymorphism, we should consider +making it easier to parameterize protocols on whether their operations +can throw. This would allow the writing of generic algorithms, e.g. over +`Sequence`, that handle both conformances that cannot throw (like +`Array`) and those that can (like a hypothetical cloud-backed +implementation). + +However, this would be a very complex feature, yet to be designed, and +it is far out-of-scope for Swift 2.0. In the meantime, most standard +protocols will be written to not allow throwing conformances, so as to +not burden the use of common generic algorithms with spurious +error-handling code. + +### Statement-like functions + +Some functions are designed to take trailing closures that feel like +sub-statements. For example, `autoreleasepool` can be used this way: + + autoreleasepool { + foo() + } + +The error-handling model doesn't cause major problems for this. The +compiler can infer that the closure throws, and `autoreleasepool` can be +overloaded on whether its argument closure throws; the overload that +takes a throwing closures would itself throw. + +There is one minor usability problem here, though. If the closure +contains throwing expressions, those expression must be explicitly +marked within the closure with `try`. However, from the compiler's +perspective, the call to `autoreleasepool` is also a call that can +throw, and so it must also be marked with `try`: + + try autoreleasepool { // 'try' is required here... + let string = try parseString() // ...and here. + ... + } + +This marking feels redundant. We want functions like `autoreleasepool` +to feel like statements, but marks inside builtin statements like `if` +don't require the outer statement to be marked. It would be better if +the compiler didn't require the outer `try`. + +On the other hand, the "statement-like" story already has a number of +other holes: for example, `break`, `continue`, and `return` behave +differently in the argument closure than in statements. In the future, +we may consider fixing that; that fix will also need to address the +error-propagation problem. + +### `using` + +A `using` statement would acquire a resource, holds it for a fixed +period of time, optionally binds it to a name, and then releases it +whenever the controlled statement exits. `using` has many similarities +to `defer`. It does not subsume `defer`, which is useful for many ad-hoc +and tokenless clean-ups. But it could be convenient for the common +pattern of a type-directed clean-up. + +### Automatically importing CoreFoundation and C functions + +CF APIs use `CFErrorRef` pretty reliably, but there are several problems +here: 1) the memory management rules for CFErrors are unclear and +potentially inconsistent. 2) we need to know when an error is raised. + +In principle, we could import POSIX functions into Swift as throwing +functions, filling in the error from `errno`. It's nearly impossible to +imagine doing this with an automatic import rule, however; much more +likely, we'd need to wrap them all in an overlay. + +In both cases, it is possible to pull these into the Swift error +handling model, but because this is likely to require massive SDK +annotations it is considered out of scope for iOS 9/OSX 10.11 & Swift +2.0. + +### Unexpected and universal errors + +As discussed above, we believe that we can extend our current model to +support untyped propagation for universal errors. Doing this well, and +in particular doing it without completely sacrificing code size and +performance, will take a significant amount of planning and insight. For +this reason, it is considered well out of scope for Swift 2.0. diff --git a/docs/ErrorHandling.rst b/docs/ErrorHandling.rst deleted file mode 100644 index 0b2424a25d571..0000000000000 --- a/docs/ErrorHandling.rst +++ /dev/null @@ -1,740 +0,0 @@ -Error Handling in Swift 2.0 -=========================== - -As a tentpole feature for Swift 2.0, we are introducing a new -first-class error handling model. This feature provides standardized -syntax and language affordances for throwing, propagating, catching, -and manipulating recoverable error conditions. - -Error handling is a well-trod path, with many different approaches in -other languages, many of them problematic in various ways. We believe -that our approach provides an elegant solution, drawing on the lessons -we've learned from other languages and fixing or avoiding some of the -pitfalls. The result is expressive and concise while still feeling -explicit, safe, and familiar; and we believe it will work beautifully -with the Cocoa APIs. - -We're intentionally not using the term "exception handling", which -carries a lot of connotations from its use in other languages. Our -proposal has some similarities to the exceptions systems in those -languages, but it also has a lot of important differences. - -Kinds of Error --------------- - -What exactly is an "error"? There are many possible error conditions, -and they don't all make sense to handle in exactly the same way, -because they arise in different circumstances and programmers have to -react to them differently. - -We can break errors down into four categories, in increasing order of -severity: - -A **simple domain error** arises from an operation that can fail in -some obvious way and which is often invoked speculatively. Parsing an -integer from a string is a really good example. The client doesn't -need a detailed description of the error and will usually want to -handle the error immediately. These errors are already well-modeled -by returning an optional value; we don't need a more complex language -solution for them. - -A **recoverable error** arises from an operation which can fail in -complex ways, but whose errors can be reasonably anticipated in -advance. Examples including opening a file or reading from a network -connection. These are the kinds of errors that Apple's APIs use -NSError for today, but there are close analogues in many other APIs, -such as ``errno`` in POSIX. - -Ignoring this kind of error is usually a bad idea, and it can even be -dangerous (e.g. by introducing a security hole). Developers should be -strongly encouraged to write code that handles the error. It's common -for developers to want to handle errors from different operations in -the same basic way, either by reporting the error to the user or -passing the error back to their own clients. - -These errors will be the focus on this proposal. - -The final two classes of error are outside the scope of this proposal. -A **universal error** is theoretically recoverable, but by its nature -the language can't help the programmer anticipate where it will come -from. A **logic failure** arises from a programmer mistake and should -not be recoverable at all. In our system, these kinds of errors are -reported either with Objective-C/C++ exceptions or simply by -logging a message and calling ``abort()``. Both kinds of error are -discussed extensively in the rationale. Having considered them -carefully, we believe that we can address them in a later release -without significant harm. - -Aspects of the Design ---------------------- - -This approach proposed here is very similar to the error handling -model manually implemented in Objective-C with the ``NSError`` -convention. Notably, the approach preserves these advantages of this -convention: - -- Whether a method produces an error (or not) is an explicit part of - its API contract. - -- Methods default to *not* producing errors unless they are explicitly - marked. - -- The control flow within a function is still mostly explicit: a - maintainer can tell exactly which statements can produce an error, - and a simple inspection reveals how the function reacts to the - error. - -- Throwing an error provides similar performance to allocating an - error and returning it -- it isn't an expensive, table-based stack - unwinding process. - -- Cocoa APIs using standard ``NSError`` patterns can be imported into - this world automatically. Other common patterns (e.g. ``CFError``, - ``errno``) can be added to the model in future versions of Swift. - -In addition, we feel that this design improves on Objective-C's error -handling approach in a number of ways: - -- It eliminates a lot of boilerplate control-flow code for propagating - errors. - -- The syntax for error handling will feel familiar to people used to - exception handling in other languages. - -- Defining custom error types is simple and ties in elegantly with - Swift enums. - -As to basic syntax, we decided to stick with the familiar language of -exception handling. We considered intentionally using different terms -(like ``raise`` / ``handle``) to try to distinguish our approach from -other languages. However, by and large, error propagation in this -proposal works like it does in exception handling, and people are -inevitably going to make the connection. Given that, we couldn't find -a compelling reason to deviate from the ``throw`` / ``catch`` legacy. - -This document just contains the basic proposal and will be very -light on rationale. We considered many different languages and -programming environments as part of making this proposal, and there's -an extensive discussion of them in the separate rationale document. -For example, that document explains why we don't simply allow all -functions to throw, why we don't propagate errors using simply an -``ErrorOr`` return type, and why we don't just make error propagation -part of a general monad feature. We encourage you to read that -rationale if you're interested in understanding why we made the -decisions we did. - -With that out of the way, let's get to the details of the proposal. - -Typed propagation ------------------ - -Whether a function can throw is part of its type. This applies to all -functions, whether they're global functions, methods, or closures. - -By default, a function cannot throw. The compiler statically enforces -this: anything the function does which can throw must appear in a -context which handles all errors. - -A function can be declared to throw by writing ``throws`` on the -function declaration or type:: - - func foo() -> Int { // This function is not permitted to throw. - func bar() throws -> Int { // This function is permitted to throw. - -``throws`` is written before the arrow to give a sensible and consistent -grammar for function types and implicit ``()`` result types, e.g.:: - - func baz() throws { - - // Takes a 'callback' function that can throw. - // 'fred' itself can also throw. - func fred(callback: (UInt8) throws -> ()) throws { - - // These are distinct types. - let a : () -> () -> () - let b : () throws -> () -> () - let c : () -> () throws -> () - let d : () throws -> () throws -> () - -For curried functions, ``throws`` only applies to the innermost -function. This function has type ``(Int) -> (Int) throws -> Int``:: - - func jerry(i: Int)(j: Int) throws -> Int { - -``throws`` is tracked as part of the type system: a function value -must also declare whether it can throw. Functions that cannot throw -are a subtype of functions that can, so you can use a function that -can't throw anywhere you could use a function that can:: - - func rachel() -> Int { return 12 } - func donna(generator: () throws -> Int) -> Int { ... } - - donna(rachel) - -The reverse is not true, since the caller would not be prepared to -handle the error. - -A call to a function which can throw within a context that is not -allowed to throw is rejected by the compiler. - -It isn't possible to overload functions solely based on whether the -functions throw. That is, this is not legal:: - - func foo() { - func foo() throws { - -A throwing method cannot override a non-throwing method or satisfy a -non-throwing protocol requirement. However, a non-throwing method can -override a throwing method or satisfy a throwing protocol requirement. - -It is valuable to be able to overload higher-order functions based on -whether an argument function throws, so this is allowed:: - - func foo(callback: () throws -> Bool) { - func foo(callback: () -> Bool) { - -``rethrows`` -~~~~~~~~~~~~ - -Functions which take a throwing function argument (including as an -autoclosure) can be marked as ``rethrows``:: - - extension Array { - func map(fn: ElementType throws -> U) rethrows -> [U] - } - -It is an error if a function declared ``rethrows`` does not include a -throwing function in at least one of its parameter clauses. - -``rethrows`` is identical to ``throws``, except that the function -promises to only throw if one of its argument functions throws. - -More formally, a function is *rethrowing-only* for a function *f* if: - -- it is a throwing function parameter of *f*, - -- it is a non-throwing function, or - -- it is implemented within *f* (i.e. it is either *f* or a function or - closure defined therein) and it does not throw except by either: - - - calling a function that is rethrowing-only for *f* or - - - calling a function that is ``rethrows``, passing only functions - that are rethrowing-only for *f*. - -It is an error if a ``rethrows`` function is not rethrowing-only for -itself. - -A ``rethrows`` function is considered to be a throwing function. -However, a direct call to a ``rethrows`` function is considered to not -throw if it is fully applied and none of the function arguments can -throw. For example:: - - // This call to map is considered not to throw because its - // argument function does not throw. - let absolutePaths = paths.map { "/" + $0 } - - // This call to map is considered to throw because its - // argument function does throw. - let streams = try absolutePaths.map { try InputStream(filename: $0) } - -For now, ``rethrows`` is a property of declared functions, not of -function values. Binding a variable (even a constant) to a function -loses the information that the function was ``rethrows``, and calls to -it will use the normal rules, meaning that they will be considered to -throw regardless of whether a non-throwing function is passed. - -For the purposes of override and conformance checking, ``rethrows`` -lies between ``throws`` and non-``throws``. That is, an ordinary -throwing method cannot override a ``rethrows`` method, which cannot -override a non-throwing method; but an ordinary throwing method can be -overridden by a ``rethrows`` method, which can be overridden by a -non-throwing method. Equivalent rules apply for protocol conformance. - -Throwing an error ------------------ - -The ``throw`` statement begins the propagation of an error. It always -take an argument, which can be any value that conforms to the -``ErrorType`` protocol (described below). - -:: - - if timeElapsed > timeThreshold { - throw HomeworkError.Overworked - } - - throw NSError(domain: "whatever", code: 42, userInfo: nil) - -As mentioned above, attempting to throw an error out of a function not -marked ``throws`` is a static compiler error. - -Catching errors ---------------- - -A ``catch`` clause includes an optional pattern that matches the -error. This pattern can use any of the standard pattern-matching -tools provided by ``switch`` statements in Swift, including boolean -``where`` conditions. The pattern can be omitted; if so, a ``where`` -condition is still permitted. If the pattern is omitted, or if it -does not bind a different name to the error, the name ``error`` is -automatically bound to the error as if with a ``let`` pattern. - -The ``try`` keyword is used for other purposes which it seems to fit far -better (see below), so ``catch`` clauses are instead attached to a -generalized ``do`` statement:: - - // Simple do statement (without a trailing while condition), - // just provides a scope for variables defined inside of it. - do { - let x = foo() - } - - // do statement with two catch clauses. - do { - ... - - } catch HomeworkError.Overworked { - // a conditionally-executed catch clause - - } catch _ { - // a catch-all clause. - } - -As with ``switch`` statements, Swift makes an effort to understand -whether catch clauses are exhaustive. If it can determine it is, then -the compiler considers the error to be handled. If not, the error -automatically propagates out out of scope, either to a lexically -enclosing ``catch`` clause or out of the containing function (which must -be marked ``throws``). - -We expect to refine the ``catch`` syntax with usage experience. - -``ErrorType`` -------------- - -The Swift standard library will provide ``ErrorType``, a protocol with -a very small interface (which is not described in this proposal). The -standard pattern should be to define the conformance of an ``enum`` to -the type:: - - enum HomeworkError : ErrorType { - case Overworked - case Impossible - case EatenByCat(Cat) - case StopStressingMeWithYourRules - } - -The ``enum`` provides a namespace of errors, a list of possible errors -within that namespace, and optional values to attach to each option. - -Note that this corresponds very cleanly to the ``NSError`` model of an -error domain, an error code, and optional user data. We expect to -import system error domains as enums that follow this approach and -implement ``ErrorType``. ``NSError`` and ``CFError`` themselves will also -conform to ``ErrorType``. - -The physical representation (still being nailed down) will make it -efficient to embed an ``NSError`` as an ``ErrorType`` and vice-versa. It -should be possible to turn an arbitrary Swift ``enum`` that conforms to -``ErrorType`` into an ``NSError`` by using the qualified type name as the -domain key, the enumerator as the error code, and turning the payload -into user data. - -Automatic, marked, propagation of errors ----------------------------------------- - -Once an error is thrown, Swift will automatically propagate it out of -scopes (that permit it), rather than relying on the programmer to -manually check for errors and do their own control flow. This is just -a lot less boilerplate for common error handling tasks. However, -doing this naively would introduce a lot of implicit control flow, -which makes it difficult to reason about the function's behavior. -This is a serious maintenance problem and has traditionally been a -considerable source of bugs in languages that heavily use exceptions. - -Therefore, while Swift automatically propagates errors, it requires -that statements and expressions that can implicitly throw be marked -with the ``try`` keyword. For example:: - - func readStuff() throws { - // loadFile can throw an error. If so, it propagates out of readStuff. - try loadFile("mystuff.txt") - - // This is a semantic error; the 'try' keyword is required - // to indicate that it can throw. - var y = stream.readFloat() - - // This is okay; the try covers the entire statement. - try y += stream.readFloat() - - // This try applies to readBool(). - if try stream.readBool() { - // This try applies to both of these calls. - let x = try stream.readInt() + stream.readInt() - } - - if let err = stream.getOutOfBandError() { - // Of course, the programmer doesn't have to mark explicit throws. - throw err - } - } - -Developers can choose to "scope" the ``try`` very tightly by writing it -within parentheses or on a specific argument or list element:: - - // Ok. - let x = (try stream.readInt()) + (try stream.readInt()) - - // Semantic error: the try only covers the parenthesized expression. - let x2 = (try stream.readInt()) + stream.readInt() - - // The try applies to the first array element. Of course, the - // developer could cover the entire array by writing the try outside. - let array = [ try foo(), bar(), baz() ] - -Some developers may wish to do this to make the specific throwing -calls very clear. Other developers may be content with knowing that -something within a statement can throw. The compiler's fixit hints will -guide developers towards inserting a single ``try`` that covers the entire -statement. This could potentially be controlled someday by a coding -style flag passed to the compiler. - -``try!`` -~~~~~~~~ - -To concisely indicate that a call is known to not actually throw at -runtime, ``try`` can be decorated with ``!``, turning the error check -into a runtime assertion that the call does not throw. - -For the purposes of checking that all errors are handled, a ``try!`` -expression is considered to handle any error originating from within -its operand. - -``try!`` is otherwise exactly like ``try``: it can appear in exactly -the same positions and doesn't affect the type of an expression. - -Manual propagation and manipulation of errors ---------------------------------------------- - -Taking control over the propagation of errors is important for some -advanced use cases (e.g. transporting an error result across threads -when synchronizing a future) and can be more convenient or natural for -specific use cases (e.g. handling a specific call differently within a -context that otherwise allows propagation). - -As such, the Swift standard library should provide a standard -Rust-like ``Result`` enum, along with API for working with it, -e.g.: - -- A function to evaluate an error-producing closure and capture the - result as a ``Result``. - -- A function to unpack a ``Result`` by either returning its - value or propagating the error in the current context. - -This is something that composes on top of the basic model, but that -has not been designed yet and details aren't included in this -proposal. - -The name ``Result`` is a stand-in and needs to be designed and -reviewed, as well as the basic operations on the type. - -``defer`` ---------- - -Swift should provide a ``defer`` statement that sets up an *ad hoc* -clean-up action to be run when the current scope is exited. This -replicates the functionality of a Java-style ``finally``, but more -cleanly and with less nesting. - -This is an important tool for ensuring that explicitly-managed -resources are released on all paths. Examples include closing a -network connection and freeing memory that was manually allocated. It -is convenient for all kinds of error-handling, even manual propagation -and simple domain errors, but is especially nice with automatic -propagation. It is also a crucial part of our long-term vision for -universal errors. - -``defer`` may be followed by an arbitrary statement. The compiler -should reject a ``defer`` action that might terminate early, whether by -throwing or with ``return``, ``break``, or ``continue``. - -Example:: - - if exists(filename) { - let file = open(filename, O_READ) - defer close(file) - - while let line = try file.readline() { - ... - } - - // close occurs here, at the end of the formal scope. - } - -If there are multiple defer statements in a scope, they are guaranteed -to be executed in reverse order of appearance. That is:: - - let file1 = open("hello.txt") - defer close(file1) - let file2 = open("world.txt") - defer close(file2) - ... - // file2 will be closed first. - -A potential extension is to provide a convenient way to mark that a -defer action should only be taken if an error is thrown. This is a -convenient shorthand for controlling the action with a flag. We will -evaluate whether adding complexity to handle this case is justified -based on real-world usage experience. - -Importing Cocoa ---------------- - -If possible, Swift's error-handling model should transparently work -with the SDK with a minimal amount of effort from framework owners. - -We believe that we can cover the vast majority of Objective-C APIs -with ``NSError**`` out-parameters by importing them as ``throws`` and -removing the error clause from their signature. That is, a method -like this one from ``NSAttributedString``:: - - - (NSData *)dataFromRange:(NSRange)range - documentAttributes:(NSDictionary *)dict - error:(NSError **)error; - -would be imported as:: - - func dataFromRange(range: NSRange, - documentAttributes dict: NSDictionary) throws -> NSData - -There are a number of cases to consider, but we expect that most can -be automatically imported without extra annotation in the SDK, by -using a couple of simple heuristics: - -* The most common pattern is a ``BOOL`` result, where a false value - means an error occurred. This seems unambiguous. - -* Also common is a pointer result, where a ``nil`` result usually - means an error occurred. This appears to be universal in - Objective-C; APIs that can return ``nil`` results seem to do so via - out-parameters. So it seems to be safe to make a policy decision - that it's okay to assume that a ``nil`` result is an error by - default. - - If the pattern for a method is that a ``nil`` result means it produced - an error, then the result can be imported as a non-optional type. - -* A few APIs return ``void``. As far as I can tell, for all of these, - the caller is expected to check for a non-``nil`` error. - -For other sentinel cases, we can consider adding a new clang attribute -to indicate to the compiler what the sentinel is: - -* There are several APIs returning ``NSInteger`` or ``NSUInteger``. At - least some of these return 0 on error, but that doesn't seem like a - reasonable general assumption. - -* ``AVFoundation`` provides a couple methods returning - ``AVKeyValueStatus``. These produce an error if the API returned - ``AVKeyValueStatusFailed``, which, interestingly enough, is not the - zero value. - -The clang attribute would specify how to test the return value for an -error. For example:: - - + (NSInteger)writePropertyList:(id)plist - toStream:(NSOutputStream *)stream - format:(NSPropertyListFormat)format - options:(NSPropertyListWriteOptions)opt - error:(out NSError **)error - NS_ERROR_RESULT(0); - - - (AVKeyValueStatus)statusOfValueForKey:(NSString *)key - error:(NSError **) - NS_ERROR_RESULT(AVKeyValueStatusFailed); - -We should also provide a Clang attribute which specifies that the -correct way to test for an error is to check the out-parameter. Both -of these attributes could potentially be used by the static analyzer, -not just Swift. (For example, they could try to detect an invalid -error check.) - -Cases that do not match the automatically imported patterns and that -lack an attribute would be left unmodified (i.e., they'd keep their -NSErrorPointer argument) and considered "not awesome" in the SDK -auditing tool. These will still be usable in Swift: callers will get -the NSError back like they do today, and have to throw the result -manually. - -For initializers, importing an initializer as throwing takes -precedence over importing it as failable. That is, an imported -initializer with a nullable result and an error parameter would be -imported as throwing. Throwing initializers have very similar -constraints to failable initializers; in a way, it's just a new axis -of failability. - -One limitation of this approach is that we need to be able to reconstruct -the selector to use when an overload of a method is introduced. For this -reason, the import is likely to be limited to methods where the error -parameter is the last one and the corresponding selector -chunk is either ``error:`` or the first chunk (see below). Empirically, -this seems to do the right thing for all but two sets of APIs in the -public API: - -* The ``ISyncSessionDriverDelegate`` category on ``NSObject`` declares - half-a-dozen methods like this:: - - - (BOOL)sessionDriver:(ISyncSessionDriver *)sender - didRegisterClientAndReturnError:(NSError **)outError; - - Fortunately, these delegate methods were all deprecated in Lion, and - are thus unavailable in Swift. - -* ``NSFileCoordinator`` has half a dozen methods where the ``error:`` - clause is second-to-last, followed by a block argument. These - methods are not deprecated as far as I know. - -The above translation rule would import methods like this one from -``NSDocument``:: - - - (NSDocument *)duplicateAndReturnError:(NSError **)outError; - -like so:: - - func duplicateAndReturnError() throws -> NSDocument - -The ``AndReturnError`` bit is common but far from universal; consider -this method from ``NSManagedObject``:: - - - (BOOL)validateForDelete:(NSError **)error; - -This would be imported as:: - - func validateForDelete() throws - -This is a really nice import, and it's somewhat unfortunate that we -can't import ``duplicateAndReturnError:`` as ``duplicate()``. - - -Potential future extensions to this model ------------------------------------------ - -We believe that the proposal above is sufficient to provide a huge -step forward in error handling in Swift programs, but there is always -more to consider in the future. Some specific things we've discussed -(and may come back to in the future) but don't consider to be core to -the Swift 2.0 model are: - -Higher-order polymorphism -~~~~~~~~~~~~~~~~~~~~~~~~~ - -We should make it easy to write higher-order functions that behave -polymorphically with respect to whether their arguments throw. This -can be done in a fairly simple way: a function can declare that it -throws if any of a set of named arguments do. As an example (using -strawman syntax):: - - func map(array: [T], fn: T -> U) throwsIf(fn) -> [U] { - ... - } - -There's no need for a more complex logical operator than disjunction -for normal higher-order stuff. - -This feature is highly desired (e.g. it would allow many otherwise -redundant overloads to be collapsed into a single definition), but it -may or may not make it into Swift 2.0 based on schedule limitations. - -Generic polymorphism -~~~~~~~~~~~~~~~~~~~~ - -For similar reasons to higher-order polymorphism, we should consider -making it easier to parameterize protocols on whether their operations -can throw. This would allow the writing of generic algorithms, e.g. -over ``Sequence``, that handle both conformances that cannot throw (like -``Array``) and those that can (like a hypothetical cloud-backed -implementation). - -However, this would be a very complex feature, yet to be designed, and -it is far out-of-scope for Swift 2.0. In the meantime, most standard -protocols will be written to not allow throwing conformances, so as to -not burden the use of common generic algorithms with spurious -error-handling code. - -Statement-like functions -~~~~~~~~~~~~~~~~~~~~~~~~ - -Some functions are designed to take trailing closures that feel like -sub-statements. For example, ``autoreleasepool`` can be used this way:: - - autoreleasepool { - foo() - } - -The error-handling model doesn't cause major problems for this. The -compiler can infer that the closure throws, and ``autoreleasepool`` -can be overloaded on whether its argument closure throws; the -overload that takes a throwing closures would itself throw. - -There is one minor usability problem here, though. If the closure -contains throwing expressions, those expression must be explicitly -marked within the closure with ``try``. However, from the compiler's -perspective, the call to ``autoreleasepool`` is also a call that -can throw, and so it must also be marked with ``try``:: - - try autoreleasepool { // 'try' is required here... - let string = try parseString() // ...and here. - ... - } - -This marking feels redundant. We want functions like -``autoreleasepool`` to feel like statements, but marks inside builtin -statements like ``if`` don't require the outer statement to be marked. -It would be better if the compiler didn't require the outer ``try``. - -On the other hand, the "statement-like" story already has a number of -other holes: for example, ``break``, ``continue``, and ``return`` -behave differently in the argument closure than in statements. In the -future, we may consider fixing that; that fix will also need to -address the error-propagation problem. - -``using`` -~~~~~~~~~ - -A ``using`` statement would acquire a resource, holds it for a fixed -period of time, optionally binds it to a name, and then releases it -whenever the controlled statement exits. ``using`` has many -similarities to ``defer``. It does not subsume ``defer``, which is useful -for many ad-hoc and tokenless clean-ups. But it could be convenient -for the common pattern of a type-directed clean-up. - -Automatically importing CoreFoundation and C functions -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -CF APIs use ``CFErrorRef`` pretty reliably, but there are several -problems here: 1) the memory management rules for CFErrors are unclear -and potentially inconsistent. 2) we need to know when an error is -raised. - -In principle, we could import POSIX functions into Swift as throwing -functions, filling in the error from ``errno``. It's nearly impossible -to imagine doing this with an automatic import rule, however; much -more likely, we'd need to wrap them all in an overlay. - -In both cases, it is possible to pull these into the Swift error -handling model, but because this is likely to require massive SDK -annotations it is considered out of scope for iOS 9/OSX 10.11 & Swift 2.0. - -Unexpected and universal errors -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -As discussed above, we believe that we can extend our current model to -support untyped propagation for universal errors. Doing this well, -and in particular doing it without completely sacrificing code size -and performance, will take a significant amount of planning and -insight. For this reason, it is considered well out of scope for -Swift 2.0. - diff --git a/docs/ErrorHandlingRationale.md b/docs/ErrorHandlingRationale.md new file mode 100644 index 0000000000000..5524729f03b90 --- /dev/null +++ b/docs/ErrorHandlingRationale.md @@ -0,0 +1,1902 @@ +Error Handling Rationale and Proposal +===================================== + +This paper surveys the error-handling world, analyzes various ideas +which have been proposed or are in practice in other languages, and +ultimately proposes an error-handling scheme for Swift together with +import rules for our APIs. + +Fundamentals +------------ + +I need to establish some terminology first. + +### Kinds of propagation + +I've heard people talk about **explicit vs. implicit propagation**. I'm +not going to use those terms, because they're not helpful: there are at +least three different things about error-handling that can be more or +less explicit, and some of the other dimensions are equally important. + +The most important dimensions of variation are: + +- Whether the language allows functions to be designated as producing + errors or not; such a language has **typed propagation**. +- Whether, in a language with typed propagation, the default rule is + that that a function can produce an error or that it can't; this is + the language's **default propagation rule**. +- Whether, in a language with typed propagation, the language enforces + this statically, so that a function which cannot produce an error + cannot call a function which can without handling it; such a + language has **statically-enforced typed propagation**. (A language + could instead enforce this dynamically by automatically inserting + code to assert if an error propagates out. C++ does this.) +- Whether the language requires all potential error sites to be + identifiable as potential error sites; such a language has **marked + propagation**. +- Whether propagation is done explicitly with the normal data-flow and + control-flow tools of the language; such a language has **manual + propagation**. In contrast, a language where control implicitly + jumps from the original error site to the proper handler has + **automatic propagation**. + +### Kinds of error + +What is an error? There may be many different possible error conditions +in a program, but they can be categorized into several kinds based on +how programmers should be expected to react to them. Since the +programmer is expected to react differently, and since the language is +the tool of the programmer's reaction, it makes sense for each group to +be treated differently in the language. + +To be clear, in many cases the kind of error reflects a conscious +decision by the author of the error-producing code, and different +choices might be useful in different contexts. The example I'm going to +use of a "simple domain error" could easily be instead treated as a +"recoverable error" (if the author expected automatic propagation to be +more useful than immediate recovery) or even a "logic failure" (if the +author wished to prevent speculative use, e.g. if checking the +precondition was very expensive). + +In order of increasing severity and complexity: + +#### Simple domain errors + +A simple domain error is something like calling `String.toInt()` on a +string that isn't an integer. The operation has an obvious precondition +about its arguments, but it's useful to be able to pass other values to +test whether they're okay. The client will often handle the error +immediately. + +Conditions like this are best modeled with an optional return value. +They don't benefit from a more complex error-handling model, and using +one would make common code unnecessarily awkward. For example, +speculatively trying to parse a `String` as an integer in Java requires +catching an exception, which is far more syntactically heavyweight (and +inefficient without optimization). + +Because Swift already has good support for optionals, these conditions +do not need to be a focus of this proposal. + +#### Recoverable errors + +Recoverable errors include file-not-found, network timeouts, and similar +conditions. The operation has a variety of possible error conditions. +The client should be encouraged to recognize the possibility of the +error condition and consider the right way to handle it. Often, this +will be by aborting the current operation, cleaning up after itself if +needed, and propagating the error to a point where it can more sensibly +be handled, e.g. by reporting it to the user. + +These are the error conditions that most of our APIs use `NSError` and +`CFError` for today. Most libraries have some similar notion. This is +the focus of this proposal. + +#### Universal errors + +The difference between universal errors and ordinary recoverable errors +is less the kind of error condition and more the potential sources of +the error in the language. An error is universal if it could arise from +such a wealth of different circumstances that it becomes nearly +impracticable for the programmer to directly deal with all the sources +of the error. + +Some conditions, if they are to be brought into the scope of +error-handling, can only conceivably be dealt with as universal errors. +These include: + +- Asynchronous conditions, like the process receiving a `SIGINT`, or + the current thread being cancelled by another thread. These + conditions could, in principle, be delivered at an arbitrary + instruction boundary, and handling them appropriately requires + extraordinary care from the programmer, the compiler, and + the runtime. +- Ubiquitous errors, like running out of memory or overflowing the + stack, that essentially any operation can be assumed to + potentially do. + +But other kinds of error condition can essentially become universal +errors with the introduction of abstraction. Reading the size of a +collection, or reading a property of an object, is not an operation that +a programmer would normally expect to produce an error. However, if the +collection is actually backed by a database, the database query might +fail. If the user must write code as if any opaque abstraction might +produce an error, they are stuck in the world of universal errors. + +Universal errors mandate certain language approaches. Typed propagation +of universal errors is impossible, other than special cases which can +guarantee to not produce errors. Marked propagation would provoke a user +revolt. Propagation must be automatic, and the implementation must be +"zero-cost", or as near to it as possible, because checking for an error +after every single operation would be prohibitive. + +For these reasons, in our APIs, universal error conditions are usually +implemented using Objective-C exceptions, although not all uses of +Objective-C exceptions fall in this category. + +This combination of requirements means that all operations must be +implicitly "unwindable" starting from almost any call site it makes. For +the stability of the system, this unwinding process must restore any +invariants that might have been temporarily violated; but the compiler +cannot assist the programmer in this. The programmer must consciously +recognize that an error is possible while an invariant is broken, and +they must do this proactively --- that, or track it down when they +inevitably forget. This requires thinking quite rigorously about one's +code, both to foresee all the error sites and to recognize that an +important invariant is in flux. + +How much of a problem this poses depends quite a lot on the code being +written. There are some styles of programming that make it pretty +innocuous. For example, a highly functional program which +conscientiously kept mutation and side-effects to its outermost loops +would naturally have very few points where any invariants were in flux; +propagating an error out of an arbitrary place within an operation would +simply abandon all the work done up to that point. However, this happy +state falls apart quite quickly the more that mutation and other +side-effects come into play. Complex mutations cannot be trivially +reversed. Packets cannot be unsent. And it would be quite amazing for us +to assert that code shouldn't be written that way, understanding nothing +else about it. As long as programmers do face these issues, the language +has some responsibility to help them. + +Therefore, in my judgment, promoting the use of universal errors is +highly problematic. They undermine the easy comprehension of code, and +they undermine the language's ability to help the programmer reason +about errors. This design will instead focus on explicitly trackable +errors of the sort that `NSError` is used for today on Apple platforms. + +However, there are some important reasons not to rule out universal +errors completely: + +- They remain the only viable means of bringing certain error + conditions into the error-handling model, as discussed above. Of + these, most run into various objections; the most important + remaining use case is "escaping", where an unexpected implementation + of an API that was not designed to throw finds itself needing to. +- Objective-C and C++ exceptions are a legitimate interoperation + problem on any conceivable platform Swift targets. Swift must have + some sort of long-term answer for them. + +These reasons don't override the problems with universal errors. It is +inherently dangerous to implicitly volunteer functions for unwinding +from an arbitrary point. We don't want to promote this model. However, +it is certainly possible to write code that handles universal errors +correctly; and pragmatically, unwinding through most code will generally +just work. Swift could support a secondary, untyped propagation +mechanism using "zero-cost" exceptions. Code can be written carefully to +minimize the extent of implicit unwinding, e.g. by catching universal +errors immediately after calling an "escaping" API and rethrowing them +with normal typed propagation. + +However, this work is outside of the scope of Swift 2.0. We can +comfortably make this decision because doing so doesn't lock us out of +implementing it in the future: + +- We do not currently support propagating exceptions through Swift + functions, so changing `catch` to catch them as well would not be a + major compatibility break. +- With some admitted awkwardness, external exceptions can be reflected + into an `ErrorType` - like model automatically by the + catch mechanism. +- In the meanwhile, developers who must handle an Objective-C + exception can always do so by writing a stub in Objective-C to + explicitly "bridge" the exception into an `NSError` out parameter. + This isn't ideal, but it's acceptable. + +#### Logic failures + +The final category is logic failures, including out of bounds array +accesses, forced unwrap of `nil` optionals, and other kinds of +assertions. The programmer has made a mistake, and the failure should be +handled by fixing the code, not by attempting to recover dynamically. + +High-reliability systems may need some way to limp on even after an +assertion failure. Tearing down the process can be viewed as a vector +for a denial-of-service attack. However, an assertion failure might +indicate that the process has been corrupted and is under attack, and +limping on anyway may open the system up for other, more serious forms +of security breach. + +The correct handling of these error conditions is an open question and +is not a focus of this proposal. Should we decide to make them +recoverable, they will likely follow the same implementation mechanism +as universal errors, if not necessarily the same language rules. + +Analysis +-------- + +Let's take a deeper look into the different dimensions of error-handling +I laid out above. + +### Propagation methods + +At a language level, there are two basic ways an error can be propagated +from an error site to something handling it. + +The first is that it can be done with the normal evaluation, data flow, +and control flow processes of the language; let's call this **manual +propagation**. Here's a good example of manual propagation using special +return values in an imperative language, C: + +``` {.sourceCode .c} +struct object *read_object(void) { + char buffer[1024]; + ssize_t numRead = read(0, buffer, sizeof(buffer)); + if (numRead < 0) return NULL; + ... +} +``` + +Here's an example of manual propagation of an error value through +out-parameters in another imperative language, Objective-C: + +``` {.sourceCode .objc} +- (BOOL) readKeys: (NSArray**) strings error: (NSError**) err { + while (1) { + NSString *key; + if ([self readKey: &key error: err]) { + return TRUE; + } + ... + } + ... +} +``` + +Here's an example of manual propagation using an ADT in an impure +functional language, SML; it's somewhat artificial because the SML +library actually uses exceptions for this: + +``` {.sourceCode .sml} +fun read_next_cmd () = + case readline(stdin) of + NONE => NONE + | SOME line => if ... +``` + +All of these excerpts explicitly test for errors using the language's +standard tools for data flow and then explicitly bypass the evaluation +of the remainder of the function using the language's standard tools for +control flow. + +The other basic way to propagate errors is in some hidden, more +intrinsic way not directly reflected in the ordinary control flow rules; +let's call this **automatic propagation**. Here's a good example of +automatic propagation using exceptions in an imperative language, Java: + +``` {.sourceCode .java} +String next = readline(); +``` + +If `readline` encounters an error, it throws an exception; the language +then terminates scopes until it dynamically reaches a `try` statement +with a matching handler. Note the lack of any code at all implying that +this might be happening. + +The chief disadvantages of manual propagation are that it's tedious to +write and requires a lot of repetitive boilerplate. This might sound +superficial, but these are serious concerns. Tedium distracts +programmers and makes them careless; careless error-handling code can be +worse than useless. Repetitive boilerplate makes code less readable, +hurting maintainability; it occupies the programmer's time, creating +opportunity costs; it discourages handling errors *well* by making it +burdensome to handle them *at all*; and it encourages shortcuts (such as +extensive macro use) which may undermine other advantages and goals. + +The chief disadvantage of automatic propagation is that it obscures the +control flow of the code. I'll talk about this more in the next section. + +Note that automatic propagation needn't be intrinsic in a language. The +propagation is automatic if it doesn't correspond to visible constructs +in the source. This effect can be duplicated as a library with any +language facility that allows restructuring of code (e.g. with macros or +other term-rewriting facilities) or overloading of basic syntax (e.g. +Haskell mapping its `do` notation onto monads). + +Note also that multiple propagation strategies may be "in play" for any +particular program. For example, Java generally uses exceptions in its +standard libraries, but some specific APIs might opt to instead return +`null` on error for efficiency reasons. Objective-C provides a fairly +full-featured exceptions model, but the standard APIs (with a few +important exceptions) reserve them solely for unrecoverable errors, +preferring manual propagation with `NSError` out-parameters instead. +Haskell has a large number of core library functions which return +`Maybe` values to indicate success or error, but it also offers at least +two features resembling traditional, automatically-propagating +exceptions (the `ErrorT` monad transform and exceptions in the `IO` +monad). + +So, while I'm going to talk as if languages implement a single +propagation strategy, it should be understood that reality will always +be more complex. It is literally impossible to prevent programmers from +using manual propagation if they want to. Part of the proposal will +discuss using multiple strategies at once. + +### Marked propagation + +Closely related to the question of whether propagation is manual or +automatic is whether it is marked or unmarked. Let's say that a language +uses **marked propagation** if there is something *at the call site* +which indicates that propagation is possible from that point. + +To a certain extent, every language using manual propagation uses marked +propagation, since the manual code to propagate the error approximately +marks the call which generated the error. However, it is possible for +the propagation logic to get separated from the call. + +Marked propagation is at odds with one other major axis of language +design: a language can't solely use marked propagation if it ever +performs implicit operations that can produce errors. For example, a +language that wanted out-of-memory conditions to be recoverable errors +would have to consider everything that could allocate memory to a source +of propagation; in a high-level language, that would include a large +number of implicit operations. Such a language could not claim to use +marked propagation. + +The reason this all matters is because unmarked propagation is a pretty +nasty thing to end up with; it makes it impossible to directly see what +operations can produce errors, and therefore to directly understand the +control flow of a function. This leaves you with two options as a +programmer: + +- You can carefully consider the actual dynamic behavior of every + function called by your function. +- You can carefully arrange your function so that there are no + critical sections where an universal error can leave things in an + unwanted state. + +There are techniques for making the second more palatable. Chiefly, they +involve never writing code that relies on normal control flow to +maintain invariants and clean up after an operation; for example, always +using constructors and destructors in C++ to manage resources. This is +compulsory in C++ with exceptions enabled because of the possibility of +implicit code that can throw, but it could theoretically be used in +other languages. However, it still requires a fairly careful and +rigorous style of programming. + +It is possible to imagine a marked form of automatic propagation, where +the propagation itself is implicit except that (local) origination +points have to be explicitly marked. This is part of our proposal, and +I'll discuss it below. + +### Typed propagation + +The next major question is whether error propagation is explicitly +tracked and limited by the language. That is, is there something +explicitly *in the declaration of a function* that tells the programmer +whether it can produce errors? Let's call this **typed propagation**. + +#### Typed manual propagation + +Whether propagation is typed is somewhat orthogonal to whether it's +manual or marked, but there are some common patterns. The most dominant +forms of manual propagation are all typed, since they pass the failure +out of the callee, either as a direct result or in an out-parameter. + +Here's another example of an out-parameter: + +``` {.sourceCode .objc} +- (instancetype)initWithContentsOfURL:(NSURL *)url encoding:(NSStringEncoding)enc error:(NSError **)error; +``` + +Out-parameters have some nice advantages. First, they're a reliable +source of marking; even if the actual propagation gets separated from +the call, you can always detect a call that can generate errors as long +as its out-parameter has a recognizable name. Second, some of the +boilerplate can be shared, because you can use the same variable as an +out-parameter multiple times; unfortunately, you can't use this to +"cheat" and only check for an error once unless you have some +conventional guarantee that later calls won't spuriously overwrite the +variable. + +A common alternative in functional languages is to return an `Either` +type: + + trait Writer { + fn write_line(&mut self, s: &str) -> Result<(), IoError>; + } + +This forces the caller to deal with the error if they want to use the +result. This works well unless the call does not really have a +meaningful result (as `write_line` does not); then it depends on whether +language makes it easy to accidentally ignore results. It also tends to +create a lot of awkward nesting: + + fn parse_two_ints_and_add_them() { + match parse_int() { + Err e => Err e + Ok x => match parse_int() { + Err e => Err e + Ok y => Ok (x + y) + } + } + } + +Here, another level of nesting is required for every sequential +computation that can fail. Overloaded evaluation syntax like Haskell's +`do` notation would help with both of these problems, but only by +switching to a kind of automatic propagation. + +Manual propagation can be untyped if it occurs through a side channel. +For example, consider an object which set a flag on itself when it +encountered an error instead of directly returning it; or consider a +variant of POSIX which expected you to separately check `errno` to see +if a particular system call failed. + +#### Typed automatic propagation + +Languages with typed automatic propagation vary along several +dimensions. + +##### The default typing rule + +The most important question is whether you opt in to producing errors or +opt out of them. That is, is a function with no specific annotation able +to produce errors or not? + +The normal resilience guideline is that you want the lazier option to +preserve more flexibility for the implementation. A function that can +produce errors is definitely more flexible, since it can do more things. +Contrariwise, changing a function that doesn't produce errors into a +function that does clearly changes its contract in ways that callers +need to respond to. Unfortunately, this has some unpleasant +consequences: + +- Marked propagation would become very burdensome. Every call would + involve an annotation, either on the function (to say it cannot + generate errors) or on the call site (to mark propagation). Users + would likely rebel against this much bookkeeping. +- Most functions cannot generate recoverable errors in the way I've + defined that. That is, ignoring sources of universal errors, most + functions can be reasonably expected to not be able to + produce errors. But if that's not the default state, that means that + most functions would need annotations; again, that's a lot of + tedious bookkeeping. It's also a lot of clutter in the API. +- Suppose that you notice that a function incorrectly lacks + an annotation. You go to fix it, but you can't without annotating + all of the functions it calls, ad infinitum; like `const` + correctness in C++, the net effect is to punish conscientious users + for trying to improve their code. +- A model which pretends that every function is a source of errors is + likely to be overwhelming for humans. Programmers ought to think + rigorously about their code, but expecting them to also make + rigorous decisions about all the code their code touches is probably + too much. Worse, without marked propagation, the compiler can't + really help the programmer concentrate on the known-possible sources + of error. +- The compiler's analysis for code generation has to assume that all + sorts of things can produce errors when they really can't. This + creates a lot of implicit propagation paths that are actually 100% + dead, which imposes a serious code-size penalty. + +The alternative is to say that, by default, functions are not being able +to generate errors. This agrees with what I'm assuming is the most +common case. In terms of resilience, it means expecting users to think +more carefully about which functions can generate errors before +publishing an API; but this is similar to how Swift already asks them to +think carefully about types. Also, they'll have at least added the right +set of annotations for their initial implementation. So I believe this +is a reasonable alternative. + +##### Enforcement + +The next question is how to enforce the typing rules that prohibit +automatic propagation. Should it be done statically or dynamically? That +is, if a function claims to not generate errors, and it calls a function +that generates errors without handling the error, should that be a +compiler error or a runtime assertion? + +The only real benefit of dynamic enforcement is that it makes it easier +to use a function that's incorrectly marked as being able to produce +errors. That's a real problem if all functions are assumed to produce +errors by default, because the mistake could just be an error of +omission. If, however, functions are assumed to not produce errors, then +someone must have taken deliberate action that introduced the mistake. I +feel like the vastly improved static type-checking is worth some +annoyance in this case. + +Meanwhile, dynamic enforcement undermines most of the benefits of typed +propagation so completely that it's hardly worth considering. The only +benefit that really remains is that the annotation serves as meaningful +documentation. So for the rest of this paper, assume that typed +propagation is statically enforced unless otherwise indicated. + +##### Specificity + +The last question is how specific the typing should be: should a +function be able to state the specific classes of errors it produces, or +should the annotation be merely boolean? + +Experience with Java suggests that getting over-specific with exception +types doesn't really work out for the best. It's useful to be able to +recognize specific classes of error, but libraries generally want to +reserve flexibility about the exact kind of error they produce, and so +many errors just end up falling into broad buckets. Different libraries +end up with their own library-specific general error classes, and +exceptions list end up just restating the library's own dependencies or +wrapping the underlying errors in ways that loses critical information. + +#### Tradeoffs of typed propagation + +Typed propagation has a number of advantages and disadvantages, mostly +independent of whether the propagation is automatic. + +The chief advantage is that it is safer. It forces programmers to do +*something* to handle or propagate errors. That comes with some +downsides, which I'll talk about, but I see this as a fairly core static +safety guarantee. This is especially important in an environment where +shuttling operations between threads is common, since it calls out the +common situation where an error needs to propagate back to the +originating thread somehow. + +Even if we're settled on using typed propagation, we should be aware of +the disadvantages and investigate ways to ameliorate them: + +- Any sort of polymorphism gets more complicated, especially + higher-order functions. Functions which cannot generate errors are + in principle subtypes of functions which can. But: + - Composability suffers. A higher-order function must decide + whether its function argument is allowed to generate errors. If + not, the function may be significantly limiting its usability, + or at least making itself much more difficult to use with + error-generating functions. If so, passing a function that does + not may require a conversion (an awkward explicit one if using + manual propagation), and the result of the call will likely + claim to be able to generate errors when, in fact, it cannot. + This can be solved with overloads, but that's a lot of + boilerplate and redundancy, especially for calls that take + multiple functions (like the function composition operator). + - If an implicit conversion is allowed, it may need to + introduce thunks. In some cases, these thunks would be + inlineable ---except that, actually, it is quite useful for code + to be able to reverse this conversion and dynamically detect + functions that cannot actually generate errors. For example, an + algorithm might be able to avoid some unnecessary bookkeeping if + it knows that its function argument never fails. This poses some + representation challenges. +- It tends to promote decentralized error handling instead of letting + errors propagate to a level that actually knows how to handle them. + - Some programmers will always be tempted to incorrectly pepper + their code with handlers that just swallow errors instead of + correctly propagating them to the right place. This is often + worse than useless; it would often be better if the error just + propagated silently, because the result can be a system in an + inconsistent state with no record of why. Good language and + library facilities for propagating errors can help avoid this, + especially when moving actions between threads. + - There are many situations where errors are not actually possible + because the programmer has carefully restricted the input. For + example, matching `` `/[0-9]{4}/ ``{.sourceCode}\` and then + parsing the result as an integer. It needs to be convenient to + do this in a context that cannot actually propagate errors, but + the facility to do this needs to be carefully designed to + discourage use for swallowing real errors. It might be + sufficient if the facility does not actually swallow the error, + but instead causes a real failure. + - It is possible that the ease of higher-order programming in + Swift might ameliorate many of these problems by letting users + writing error-handling combinators. That is, in situations where + a lazy Java programmer would find themselves writing a + `try/catch` to swallow an exception, Swift would allow them to + do something more correct with equal convenience. + +One other minor advantage of marked, statically-enforced typed +propagation: it's a boon for certain kinds of refactoring. Specifically, +when a refactor makes an operation error-producing when it wasn't +before, the absence of any those properties makes the refactor more +treacherous and increases the odds of accidentally introducing a bug. If +propagation is untyped, or the typing isn't statically enforced, the +compiler isn't going to help you at all to find call sites which need to +have error-checking code. Even with static typed propagation, if the +propagation isn't marked specifically on the call site, the compiler +won't warn you about calls made from contexts that can handle or +implicitly propagate the error. But if all these things are true, the +compiler will force you to look at all the existing call sites +individually. + +### Error Types + +There are many kinds of error. It's important to be able to recognize +and respond to specific error causes programmatically. Swift should +support easy pattern-matching for this. + +But I've never really seen a point to coarser-grained categorization +than that; for example, I'm not sure how you're supposed to react to an +arbitrary, unknown IO error. And if there are useful error categories, +they can probably be expressed with predicates instead of public +subclasses. I think we start with a uni-type here and then challenge +people to come up with reasons why they need anything more. + +### Implementation design + +There are several different common strategies for implementing automatic +error propagation. (Manual propagation doesn't need special attention in +the implementation design.) + +The implementation has two basic tasks common to most languages: + +- Transferring control through scopes and functions to the appropriate + handler for the error. +- Performing various semantic "clean up" tasks for the scopes that + were abruptly terminated: + - tearing down local variables, like C++ variables with + destructors or strong/weak references in ARC-like languages; + - releasing heap-allocated local variables, like captured + variables in Swift or `__block` variables in ObjC; + - executing scope-specific termination code, like C\#'s `using` or + Java/ObjC's `synchronized` statements; and + - executing ad hoc cleanup blocks, like `finally` blocks in Java + or `defer` actions in Swift. + +Any particular call frame on the stack may have clean-ups or potential +handlers or both; call these **interesting frames**. + +#### Implicit manual propagation + +One strategy is to implicitly produce code to check for errors and +propagate them up the stack, imitating the code that the programmer +would have written under manual propagation. For example, a function +call could return an optional error in a special result register; the +caller would check this register and, if appropriate, unwind the stack +and return the same value. + +Since propagation and unwinding are explicit in the generated code, this +strategy hurts runtime performance along the non-error path more than +the alternatives, and more code is required to do the explicitly +unwinding. Branches involved in testing for errors are usually very easy +to predict, so in hot code the direct performance impact is quite small, +and the total impact is dominated by decreased code locality. Code can't +always be hot, however. + +These penalties are suffered even by uninteresting frames unless they +appear in tail position. (An actual tail call isn't necessary; there +just can't be anything that error propagation would skip.) And functions +must do some added setup work before returning. + +The upside is that the error path suffers no significant penalties +beyond the code-size impact. The code-size impact can be significant, +however: there is sometimes quite a lot of duplicate code needed for +propagation along the error path. + +This approach is therefore relatively even-handed about the error vs. +the non-error path, although it requires some care in order to minimize +code-size penalties for parallel error paths. + +#### `setjmp` / `longmp` + +Another strategy to is to dynamically maintain a thread-local stack of +interesting frames. A function with an interesting frame must save +information about its context in a buffer, like `setjmp` would, and then +register that buffer with the runtime. If the scope returns normally, +the buffer is accordingly unregistered. Starting propagation involves +restoring the context for the top of the interesting-frames stack; the +place where execution returns is called the "landing pad". + +The advantage of this is that uninteresting frames don't need to do any +work; context restoration just skips over them implicitly. This is +faster both for the error and non-error paths. It is also possible to +optimize this strategy so that (unlike `setjmp`) the test for an error +is implicitly elided: use a slightly different address for the landing +pad, so that propagating errors directly restore to that location. + +The downside is that saving the context and registering the frame are +not free: + +- Registering the frame requires an access to thread-local state, + which on our platforms means a function call because we're not + willing to commit to anything more specific in the ABI. +- Jumping across arbitary frames invalidates the callee-save + registers, so the registering frame must save them all eagerly. In + calling conventions with many callee-save registers, this can be + very expensive. However, this is only necessary when it's possible + to resume normal execution from the landing pad: if the landing pad + only has clean-ups and therefore always restarts propagation, those + registers will have been saved and restored further out. +- Languages like C++, ObjC ARC, and Swift that have non-trivial + clean-ups for many local variables tend to have many functions with + interesting frames. This means both that the context-saving + penalties are higher and that skipping uninteresting frames is a + less valuable optimization. +- By the same token, functions in those languages often have many + different clean-ups and/or handlers. For example, every new + non-trivial variable might introduce a new clean-up. The function + must either register a new landing pad for each clean-up + (very expensive!) or track its current position in a way that a + function-wide landing pad can figure out what scope it was in. + +This approach can be hybridized with the unwinding approach below so +that the interesting-frames stack abstractly describes the clean-ups in +the frame instead of just restoring control somewhere and expecting the +frame to figure it out. This can decrease the code size impact +significantly for the common case of frames that just need to run some +clean-ups before propagating the error further. It may even completely +eliminate the need for a landing pad. + +The ObjC/C++ exceptions system on iOS/ARM32 is kindof like that hybrid. +Propagation and clean-up code is explicit in the function, but the +registered context includes the "personality" information from the +unwinding tables, which makes the decision whether to land at the +landing pad at all. It also uses an optimized `setjmp` implementation +that both avoids some context-saving and threads the branch as described +above. + +The ObjC exceptions system on pre-modern runtimes (e.g. on PPC and i386) +uses the standard `setjmp` / `longjmp` functions. Every protected scope +saves the context separately. This is all implemented in a very unsafe +way that does not behave well in the presence of inlining. + +Overall, this approach requires a lot of work in the non-error path of +functions with interesting frames. Given that we expects functions with +interesting frames to be very common in Swift, this is not an +implementation approach we would consider in the abstract. However, it +is the implementation approach for C++/ObjC exceptions on iOS/ARM32, so +we need to at least interoperate with that. + +#### Table-based unwinding + +The final approach is side-table stack unwinding. This relies on being +able to accurately figure out how to unwind through an arbitrary +function on the system, given only the return address of a call it made +and the stack pointer at that point. + +On our system, this proceeds as follows. From an instruction pointer, +the system unwinder looks up what linked image (executable or dylib) +that function was loaded from. The linked image contains a special +section, a table of unwind tables indexed by their offset within the +linked image. Every non-leaf function should have an entry within this +table, which provides sufficient information to unwind the function from +an arbitrary call site. + +This lookup process is quite expensive, especially since it has to +repeat all the way up the stack until something actually handles the +error. This makes the error path extremely slow. However, no explicit +setup code is required along the non-error path, and so this approach is +sometimes known as "zero-cost". That's something of a misnomer, because +it does have several costs that can affect non-error performance. First, +there's a small amount of load-time work required in order to resolve +relocations to symbols used by the unwind tables. Second, the error path +often requires code in the function, which can decrease code locality +even if never executed. Third, the error path may use information that +the non-error path would otherwise discard. And finally, the unwind +tables themselves can be fairly large, although this is generally only a +binary-size concern because they are carefully arranged to not need to +be loaded off of disk unless an exception is thrown. But overall, +"zero-cost" is close enough to correct. + +To unwind a frame in this sense specifically means: + +- Deciding whether the function handles the error. +- Cleaning up any interesting scopes that need to be broken down + (either to get to the handler or to leave the function). +- If the function is being fully unwound, restoring any callee-save + registers which the function might have changed. + +This is language-specific, and so the table contains language-specific +"personality" information, including a reference to a function to +interpret it. This mechanism means that the unwinder is extremely +flexible; not only can it support arbitrary languages, but it can +support different language-specific unwinding table layouts for the same +language. + +Our current personality records for C++ and Objective-C contain just +enough information to decide (1) whether an exception is handled by the +frame and (2) if not, whether a clean-up is currently active. If either +is true, it restores the context of a landing pad, which manually +executes the clean-ups and enters the handler. This approach generally +needs as much code in the function as implicit manual propagation would. +However, we could optimize this for many common cases by causing +clean-ups to be called automatically by the interpretation function. +That is, instead of a landing pad that looks notionally like this: + + void *exception = ...; + SomeCXXType::~SomeCXXType(&foo); + objc_release(bar); + objc_release(baz); + _Unwind_Resume(exception); + +The unwind table would have a record that looks notionally like this: + + CALL_WITH_FRAME_ADDRESS(&SomeCXXType::~SomeCXXType, FRAME_OFFSET_OF(foo)) + CALL_WITH_FRAME_VALUE(&objc_release, FRAME_OFFSET_OF(bar)) + CALL_WITH_FRAME_VALUE(&objc_release, FRAME_OFFSET_OF(baz)) + RESUME + +And no code would actually be needed in the function. This would +generally slow the error path down, because the interpretation function +would have to interpret this mini-language, but it would move all the +overhead out of the function and into the error table, where it would be +more compact. + +This is something that would also benefit C++ code. + +### Clean-up actions + +Many languages have a built-in language tool for performing arbitrary +clean-up when exiting a scope. This has two benefits. The first is that, +even ignoring error propagation, it acts as a "scope guard" which +ensures that the clean-up is done if the scope is exited early due to a +`return`, `break`, or `continue` statement; otherwise, the programmer +must carefully duplicate the clean-up in all such places. The second +benefit is that it makes clean-up tractable in the face of automatic +propagation, which creates so many implicit paths of control flow out of +the scope that expecting the programmer to cover them all with explicit +catch-and-rethrow blocks would be ridiculous. + +There's an inherent tension in these language features between putting +explicit clean-up code in the order it will be executed and putting it +near the code it's cleaning up after. The former means that a +top-to-bottom read of the code tells you what actions are being +performed when; you don't have to worry about code implicitly +intervening at the end of a scope. The latter makes it easy to verify at +the point that a clean-up is needed that it will eventually happen; you +don't need to scan down to the finally block and analyze what happens +there. + +#### `finally` + +Java, Objective-C, and many other languages allow `try` statements to +take a `finally` clause. The clause is an ordinary scope and may take +arbitrary actions. The `finally` clause is performed when the preceding +controlled scopes (including any `catch` clauses) are exited in any way: +whether by falling off the end, directly branching or returning out, or +throwing an exception. + +`finally` is a rather awkward and verbose language feature. It separates +the clean-up code from the operation that required it (although this has +benefits, as discussed above). It adds a lot of braces and indentation, +so edits that add new clean-ups can require a lot of code to be +reformatted. When the same scope needs multiple clean-ups, the +programmer must either put them in the same `finally` block (and thus +create problems with clean-ups that might terminate the block early) or +stack them up in separate blocks (which can really obscure the otherwise +simple flow of code). + +#### `defer` + +Go provides a `defer` statement that just enqueues arbitrary code to be +executed when the function exits. (More details of this appear in the +survey of Go.) + +This allows the defer action to be written near the code it "balances", +allowing the reader to immediately see that the required clean-up will +be done (but this has drawbacks, as discussed above). It's very compact, +which is nice as most defer actions are short. It also allow multiple +actions to pile up without adding awkward nesting. However, the +function-exit semantics exacerbate the problem of searching for +intervening clean-up actions, and they introduce semantic and +performance problems with capturing the values of local variables. + +#### Destructors + +C++ allows types to define destructor functions, which are called when a +function goes out of scope. + +These are often used directly to clean up the ownership or other +invariants on the type's value. For example, an owning-pointer type +would free its value in its destructor, whereas a hash-table type would +destroy its entries and free its buffer. + +But they are also often used idiomatically just for the implicit +destructor call, as a "scope guard" to ensure that something is done +before the current operation completes. For an example close to my own +heart, a compiler might use such a guard when parsing a local scope to +ensure that new declarations are removed from the scope chains even if +the function exits early due to a parse error. Unfortunately, since type +destructors are C++'s only tool for this kind of clean-up, introducing +ad-hoc clean-up code requires defining a new type every time. + +The unique advantage of destructors compared to the options above is +that destructors can be tied to temporary values created during the +evaluation of an expression. + +Generally, a clean-up action becomes necessary as the result of some +"acquire" operation that occurs during an expression. `defer` and +`finally` do not take effect until the next statement is reached, which +creates an atomicity problem if code can be injected after the acquire. +(For `finally`, this assumes that the acquire appears *before* the +`try`. If instead the acquire appears *within* the `try`, there must be +something which activates the clean-up, and that has the same atomicity +problem.) + +In contrast, if the acquire operation always creates a temporary with a +destructor that does the clean-up, the language automatically guarantees +this atomicity. This pattern is called "resource acquisition is +initialization", or "RAII". Under RAII, all resources that require +clean-up are carefully encapsulated within types with user-defined +destructors, and the act of constructing an object of that type is +exactly the act of acquiring the underlying resource. + +Swift does not support user-defined destructors on value types, but it +does support general RAII-like programming with class types and `deinit` +methods, although (at the moment) the user must take special care to +keep the object alive, as Swift does not normally guarantee the +destruction order of objects. + +RAII is very convenient when there's a definable "resource" and +somebody's already wrapped its acquisition APIs to return +appropriately-destructed objects. For other tasks, where a reasonable +programmer might balk at defining a new type and possibly wrapping an +API for a single purpose, a more *ad hoc* approach may be warranted. + +Survey +------ + +### C + +C doesn't really have a consensus error-handling scheme. There's a +built-in unwinding mechanism in `setjmp` and `longjmp`, but it's +disliked for a host of good reasons. The dominant idiom in practice is +for a function to encode failure using some unreasonable value for its +result, like a null pointer or a negative count. The bad value(s) are +often function-specific, and sometimes even argument- or state-specific. + +On the caller side, it is unfortunately idiomatic (in some codebases) to +have a common label for propagating failure at the end of a function +(hence `goto fail`); this is because there's no inherent language +support for ensuring that necessary cleanup is done before propagating +out of a scope. + +### C++ + +C++ has exceptions. Exceptions can have almost any type in the language. +Propagation typing is tied only to declarations; an indirect function +pointer is generally assumed to be able to throw. Propagation typing +used to allow functions to be specific about the kinds of exceptions +they could throw (`` `throws +(std::exception) ``{.sourceCode}), but this is deprecated in favor of just indicating +whether a function can throw (:code:noexcept(false)). + +C++ aspires to making out-of-memory a recoverable condition, and so +allocation can throw. Therefore, it is essentially compulsory for the +language to assume that constructors might throw. Since constructors +are called pervasively and implicitly, it makes sense for the default +rule to be that all functions can throw. Since many error sites are +implicit, there is little choice but to use automatic unmarked +propagation. The only reasonable way to clean up after a scope in +such a world is to allow the compiler to do it automatically. C++ +programmers therefore rely idiomatically on a pattern of shifting all +scope cleanup into the destructors of local variables; sometimes such +local values are created solely to set up a cleanup action in this +way. + +Different error sites occur with a different set of cleanups active, +and there are a large number of such sites. In fact, prior to C++11, +compilers were forced to assume by default that destructor calls could +throw, so cleanups actually created more error sites. This all adds +up to a significant code-size penalty for exceptions, even in projects +which don't directly use them and which have no interest in recovering +from out-of-memory conditions. For this reason, many C++ projects +explicitly disable exceptions and rely on other error propagation +mechanisms, on which there is no widespread consensus. + +Objective C +----------- + +Objective C has a first-class exceptions mechanism which is similar in +feature set to Java's: @throw\` / `@try` / `@catch` / `@finally`. +Exception values must be instances of an Objective-C class. The language +does a small amount of implicit frame cleanup during exception +propagation: locks held by `@synchronized` are released, stack copies of +`__block` variables are torn down, and ARC `__weak` variables are +destroyed. However, the language does not release object pointers held +in local variables, even (by default) under ARC. + +Objective C exceptions used to be implemented with `setjmp`, `longjmp`, +and thread-local state managed by a runtime, but the only surviving +platform we support which does that is i386, and all others now use a +"zero-cost" implementation that interoperates with C++ exceptions. + +Objective C exceptions are *mostly* only used for unrecoverable +conditions, akin to what I called "failures" above. There are a few +major exceptions to this rule, where APIs that do use exceptions to +report errors. + +Instead, Objective C mostly relies on manual propagation, predominantly +using out-parameters of type `NSError**`. Whether the call failed is +usually *not* indicated by whether a non-`nil` error was written into +this parameter; calls are permitted both to succeed and write an error +object into the parameter (which should be ignored) and to report an +error without creating an actual error object. Instead, whether the call +failed is reported in the formal return value. The most common +convention is for a false `BOOL` result or null object result to mean an +error, but ingenious programmers have come up with many other +conventions, and there do exist APIs where a null object result is +valid. + +CF APIs, meanwhile, have their own magnificent set of somewhat +inconsistent conventions. + +Therefore, we can expect that incrementally improving CF / Objective C +interoperation is going to be a long and remarkably painful process. + +### Java + +Java has a first-class exceptions mechanism with unmarked automatic +propagation: `throw` / `try` / `catch` / `finally`. Exception values +must be instances of something inheriting from `Throwable`. Propagation +is generally typed with static enforcement, with the default being that +a call cannot throw exceptions *except* for subclasses of `Error` and +`RuntimeException`. The original intent was that these classes would be +used for catastrophic runtime errors (`Error`) and programming mistakes +caught by the runtime (`RuntimeException`), both of which we would +classify as unrecoverable failures in our scheme; essentially, Java +attempts to promote a fully statically-enforced model where truly +catastrophic problems can still be handled when necessary. +Unfortunately, these motivations don't seem to have been communicated +very well to developers, and the result is kindof a mess. + +Java allows methods to be very specific about the kinds of exception +they throw. In my experience, exceptions tend to fall into two +categories: + +- There are some very specific exception kinds that callers know to + look for and handle on specific operations. Generally these are + obvious, predictable error conditions, like a host name not + resolving, or like a string not being formatted correctly. +- There are also a lot of very vague, black-box exception kinds that + can't really be usefully responded to. For example, if a method + throws `IOException`, there's really nothing a caller can do except + propagate it and abort the current operation. + +So specific typing is useful if you can exhaustively handle a small +number of specific failures. As soon as the exception list includes any +kind of black box type, it might as well be a completely open set. + +### C\# + +C\#'s model is almost exactly like Java's except that it is untyped: all +methods are assumed to be able to throw. For this reason, it also has a +simpler type hierarchy, where all exceptions just inherit from +`Exception`. + +The rest of the hierarchy doesn't really make any sense to me. Many +things inherit directly from `Exception`, but many other things inherit +from a subclass called `SystemException`. `SystemException` doesn't seem +to be any sort of logical grouping: it includes all the +runtime-assertion exceptions, but it also includes every exception +that's thrown anywhere in the core library, including XML and IO +exceptions. + +C\# also has a `using` statement, which is useful for binding something +over a precise scope and then automatically disposing it on all paths. +It's just built on top of `try` / `finally`. + +### Haskell + +Haskell provides three different common error-propagation mechanisms. + +The first is that, like many other functional languages, it supports +manual propagation with a `Maybe` type. A function can return `None` to +indicate that it couldn't produce a more useful result. This is the most +common failure method for functions in the functional subset of the +library. + +The `IO` monad also provides true exceptions with unmarked automatic +propagation. These exceptions can only be handled as an `IO` action, but +are otherwise untyped: there is no way to indicate whether an `IO` +action can or cannot throw. Exceptions can be thrown either as an `IO` +action or as an ordinary lazy functional computation; in the latter +case, the exception is only thrown if the computation is evaluated for +some reason. + +The `ErrorT` monad transform provides typed automatic propagation. In an +amusing twist, since the only native computation of `ErrorT` is +`throwError`, and the reason to write a computation specifically in +`ErrorT` is if it's throwing, and every other computation must be +explicitly lifted into the monad, `ErrorT` effectively uses marked +propagation by omission, since everything that *can't* throw is +explicitly marked with a `lift`: + +``` {.sourceCode .haskell} +prettyPrintShiftJIS :: ShiftJISString -> ErrorT TranscodeError IO () +prettyPrintShiftJIS str = do + lift $ putChar '"' -- lift turns an IO computation into an ErrorT computation + case transcodeShiftJISToUTF8 str of + Left error -> throwError error + Right value -> lift $ putEscapedString value + lift $ putChar '"' +``` + +### Rust + +Rust distinguishes between *failures* and *panics*. + +A panic is an assertion, designed for what I called logic failures; +there's no way to recover from one, it just immediately crashes. + +A failure is just when a function doesn't produce the value you might +expect, which Rust encourages you to express with either `Option` +(for simple cases, like what I described as simple domain errors) or +`Result` (which is effectively the same, except carrying an error). +In either case, it's typed manual propagation, although Rust does at +least offer a standard macro which wraps the common +pattern-match-and-return pattern for `Result`. + +The error type in Rust is a very simple protocol, much like this +proposal suggests. + +### Go + +Go uses an error result, conventionally returned as the final result of +functions that can fail. The caller is expected to manually check +whether this is nil; thus, Go uses typed manual propagation. + +The error type in Go is an interface named `error`, with one method that +returns a string description of the error. + +Go has a `defer` statement: + + defer foo(x, y) + +The argument has to be a call (possibly a method call, possibly a call +to a closure that you made specifically to immediately call). All the +operands are evaluated immediately and captured in a deferred action. +Immediately after the function exits (through whatever means), all the +deferred actions are executed in LIFO order. Yes, this is tied to +function exit, not scope exit, so you can have a dynamic number of +deferred actions as a sort of implicit undo stack. Overall, it's a nice +if somewhat quirky way to do ad-hoc cleanup actions. + +It is also a key part of a second, funky kind of error propagation, +which is essentially untyped automatic propagation. If you call `panic` +--- and certain builtin operations like array accesses behave like they +do --- it immediately unwinds the stack, running deferred actions as it +goes. If a function's deferred action calls `recover`, the panic stops, +the rest of the deferred actions for the function are called, and the +function returns. A deferred action can write to the named results, +allowing a function to turn a panic error into a normal, final-result +error. It's conventional to not panic over API boundaries unless you +really mean it; recoverable errors are supposed to be done with +out-results. + +### Scripting languages + +Scripting languages generally all use (untyped, obviously) automatic +exception propagation, probably because it would be quite error-prone to +do manual propagation in an untyped language. They pretty much all fit +into the standard C++/Java/C\# style of `throw` / `try` / `catch`. Ruby +uses different keywords for it, though. + +I feel like Python uses exceptions a lot more than most other scripting +languages do, though. + +Proposal +-------- + +### Automatic propagation + +Swift should use automatic propagation of errors, rather than relying on +the programmer to manually check for them and return out. It's just a +lot less boilerplate for common error handling tasks. This introduces an +implicit control flow problem, but we can ameliorate that with marked +propagation; see below. + +There's no compelling reason to deviate from the `throw` / `catch` +legacy here. There are other options, like `raise` / `handle`. In +theory, switching would somewhat dissociate Swift from the legacy of +exceptions; people coming from other languages have a lot of assumptions +about exceptions which don't necessarily apply to Swift. However, our +error model is similar enough to the standard exception model that +people are inevitably going to make the connection; there's no getting +around the need to explain what we're trying to do. So using different +keywords just seems petty. + +Therefore, Swift should provide a `throw` expression. It requires an +operand of type `Error` and formally yields an arbitrary type. Its +dynamic behavior is to transfer control to the innermost enclosing +`catch` clause which is satisfied by the operand. A quick example: + + if timeElapsed() > timeThreshold { throw HomeworkError.Overworked } + +A `catch` clause includes a pattern that matches an error. We want to +repurpose the `try` keyword for marked propagation, which it seems to +fit far better, so `catch` clauses will instead be attached to a +generalized `do` statement: + + do { + ... + + } catch HomeworkError.Overworked { + // a conditionally-executed catch clause + + } catch _ { + // a catch-all clause + } + +Swift should also provide some tools for doing manual propagation. We +should have a standard Rust-like `` `Result ``{.sourceCode}\` enum in +the library, as well as a rich set of tools, e.g.: + +- A function to evaluate an error-producing closure and capture the + result as a `` `Result ``{.sourceCode}\`. +- A function to unpack a `` `Result ``{.sourceCode}\` by either + returning its value or propagating the error in the current context. +- A futures library that traffics in `` `Result ``{.sourceCode}\` + when applicable. +- An overload of `dispatch_sync` which takes an error-producing + closure and propagates an error in the current context. +- etc. + +### Typed propagation + +Swift should use statically-enforced typed propagation. By default, +functions should not be able to throw. A call to a function which can +throw within a context that is not allowed to throw should be rejected +by the compiler. + +Function types should indicate whether the function throws; this needs +to be tracked even for first-class function values. Functions which do +not throw are subtypes of functions that throw. + +This would be written with a `throws` clause on the function declaration +or type: + + // This function is not permitted to throw. + func foo() -> Int { + // Therefore this is a semantic error. + return try stream.readInt() + } + + // This function is permitted to throw. + func bar() throws -> Int { + return try stream.readInt() + } + + // ‘throws’ is written before the arrow to give a sensible and + // consistent grammar for function types and implicit () result types. + func baz() throws { + if let byte = try stream.getOOB() where byte == PROTO_RESET { + reset() + } + } + + // ‘throws’ appears in a consistent position in function types. + func fred(callback: (UInt8) throws -> ()) throws { + while true { + let code = try stream.readByte() + if code == OPER_CLOSE { return } + try callback(code) + } + } + + // It only applies to the innermost function for curried functions; + // this function has type: + // (Int) -> (Int) throws -> Int + func jerry(i: Int)(j: Int) throws -> Int { + // It’s not an error to use ‘throws’ on a function that can’t throw. + return i + j + } + +The reason to use a keyword here is that it's much nicer for function +declarations, which generally outnumber function types by at least an +order of magnitude. A punctuation mark would be easily lost or mistaken +amidst all the other punctuation in a function declaration, especially +if the punctuation mark were something like `!` that can validly appear +at the end of a parameter type. It makes sense for the keyword to appear +close to the return type, as it's essentially a part of the result and a +programmer should be able to see both parts in the same glance. The +keyword appears before the arrow for the simple reason that the arrow is +optional (along with the rest of the return type) in function and +initializer declarations; having the keyword appear in slightly +different places based on the presence of a return type would be silly +and would making adding a non-void return type feel awkward. The keyword +itself should be descriptive, and it's particularly nice for it to be a +form of the verb used by the throwing expression, conjugated as if +performed by the function itself. Thus, `throw` becomes `throws`; if we +used `raise` instead, this would be `raises`, which I personally find +unappealing for reasons I'm not sure I can put a name to. + +It shouldn't be possible to overload functions solely based on whether +the functions throw. That is, this is not legal: + + func foo() { ... } // called in contexts that cannot throw + func foo() throws { ... } // called in contexts that can throw + +It is valuable to be able to overload higher-order functions based on +whether an argument function throws; it is easy to imagine algorithms +that can be implemented more efficiently if they do not need to worry +about exceptions. (We do not, however, particularly want to encourage a +pattern of duplicating This is straightforward if the primary +type-checking pass is able to reliably decide whether a function value +can throw. + +Typed propagation checking can generally be performed in a secondary +pass over a type-checked function body: if a function is not permitted +to throw, walk its body and verify that there are no `throw` expressions +or calls to functions that can `throw`. If all throwing calls must be +marked, this can be done prior to type-checking to decide syntactically +whether a function can apparently throw; of course, the later pass is +still necessary, but the ability to do this dramatically simplifies the +implementation of the type-checker, as discussed below. Certain +type-system features may need to be curtailed in order to make this +implementation possible for schedule reasons. (It's important to +understand that this is *not* the motivation for marked propagation. +It's just a convenient consequence that marked propagation makes this +implementation possible.) + +Reliably deciding whether a function value can throw is easy for +higher-order uses of declared functions. The problem, as usual, is +anonymous functions. We don't want to require closures to be explicitly +typed as throwing or non-throwing, but the fully-accurate inference +algorithm requires a type-checked function body, and we can't always +type-check an anonymous function independently of its enclosing context. +Therefore, we will rely on being able to do a pass prior to +type-checking to syntactically infer whether a closure throws, then +making a second pass after type-checking to verify the correctness of +that inference. This may break certain kinds of reasonable code, but the +multi-pass approach should let us heuristically unbreak targeted cases. + +Typed propagation has implications for all kinds of polymorphism: + +#### Higher-order polymorphism + +We should make it easy to write higher-order functions that behave +polymorphically w.r.t. whether their arguments throw. This can be done +in a fairly simple way: a function can declare that it throws if any of +a set of named arguments do. As an example (using strawman syntax): + + func map(array: [T], fn: T throws -> U) throwsIf(fn) -> [U] { + ... + } + +There's no need for a more complex logical operator than disjunction. +You can construct really strange code where a function throws only if +one of its arguments doesn't, but it'd be contrived, and it's hard to +imagine how they could be type-checked without a vastly more +sophisticated approach. Similarly, you can construct situations where +whether a function can throw is value-dependent on some other argument, +like a "should I throw an exception" flag, but it's hard to imagine such +cases being at all important to get right in the language. This schema +is perfectly sufficient to express normal higher-order stuff. + +In fact, while the strawman syntax above allows the function to be +specific about exactly which argument functions cause the callee to +throw, that's already overkill in the overwhelmingly likely case of a +function that throws if any of its argument functions throw (and there's +probably only one). So it would probably be better to just have a single +`rethrows` annotation, with vague plans to allow it to be parameterized +in the future if necessary. + +This sort of propagation-checking would be a straightforward extension +of the general propagation checker. The normal checker sees that a +function isn't allowed to propagate out and looks for propagation +points. The conditional checker sees that a function has a conditional +propagation clause and looks for propagation points, assuming that the +listed functions don't throw (including when looking at any conditional +propagation clauses). The parameter would have to be a `let`. + +We probably do need to get higher-order polymorphism right in the first +release, because we will need it for the short-circuiting operators. + +#### Generic polymorphism + +It would be useful to be able to parameterize protocols, and protocol +conformances, on whether the operations produce errors. Lacking this +feature means that protocol authors must decide to either conservatively +allow throwing conformances, and thus force all generic code using the +protocol to deal with probably-spurious errors, or aggressively forbid +them, and thus forbid conformances by types whose operations naturally +throw. + +There are several different ways we could approach this problem, but +after some investigation I feel confident that they're workable. +Unfortunately, they are clearly out-of-scope for the first release. For +now, the standard library should provide protocols that cannot throw, +even though this limits some potential conformances. (It's worth noting +that such conformances generally aren't legal today, since they'd need +to return an error result somehow.) + +A future direction for both generic and higher-order polymorphism is to +consider error propagation to be one of many possible effects in a +general, user-extensible effect tracking system. This would allow the +type system to check that certain specific operations are only allowed +in specific contexts: for example, that a blocking operation is only +allowed in a blocking context. + +#### Error type + +The Swift standard library will provide `ErrorType`, a protocol with a +very small interface (which is not described in this proposal). The +standard pattern should be to define the conformance of an `enum` to the +type: + + enum HomeworkError : ErrorType { + case Overworked + case Impossible + case EatenByCat(Cat) + case StopStressingMeWithYourRules + } + +The `enum` provides a namespace of errors, a list of possible errors +within that namespace, and optional values to attach to each option. + +For now, the list of errors in a domain will be fixed, but permitting +future extension is just ordinary enum resilience, and the standard +techniques for that will work fine in the future. + +Note that this corresponds very cleanly to the `NSError` model of an +error domain, an error code, and optional user data. We expect to import +system error domains as enums that follow this approach and implement +`ErrorType`. `NSError` and `CFError` themselves will also conform to +`ErrorType`. + +The physical representation (still being nailed down) will make it +efficient to embed an `NSError` as an `ErrorType` and vice-versa. It +should be possible to turn an arbitrary Swift `enum` that conforms to +`ErrorType` into an `NSError` by using the qualified type name as the +domain key, the enumerator as the error code, and turning the payload +into user data. + +It's acceptable to allocate memory whenever an error is needed, but our +representation should not inhibit the optimizer from forwarding a +`throw` directly to a `catch` and removing the intermediate error +object. + +### Marked propagation + +Swift should use marked propagation: there should be some lightweight +bit of syntax decorating anything that is known be able to throw (other +than a `throw` expression itself, of course). + +Our proposed syntax is to repurpose `try` as something that can be +wrapped around an arbitrary expression: + + // This try applies to readBool(). + if try stream.readBool() { + + // This try applies to both of these calls. + let x = try stream.readInt() + stream.readInt() + + // This is a semantic error; it needs a try. + var y = stream.readFloat() + + // This is okay; the try covers the entire statement. + try y += stream.readFloat() + } + +Developers can "scope" the `try` very tightly by writing it within +parentheses or on a specific argument or list element: + + // Semantic error: the try only covers the parenthesized expression. + let x = (try stream.readInt()) + stream.readInt() + + // The try applies to the first array element. Of course, the + // developer could cover the entire array by writing the try outside. + let array = [ try foo(), bar(), baz() ] + +Some developers may wish to do this to make the specific throwing calls +very clear. Other developers may be content with knowing that something +within a statement can throw. + +We also briefly considered the possibility of putting the marker into +the call arguments clause, e.g.: + + parser.readKeys(&strings, try) + +This works as long as the only throwing calls are written syntactically +as calls; this covers calls to free functions, methods, and +initializers. However, it effectively requires Swift to forbid operators +and property and subscript accessors from throwing, which may not be a +reasonable limitation, especially for operators. It is also somewhat +unnatural, and it forces users to mark every single call site instead of +allowing them to mark everything within a statement at once. + +Autoclosures pose a problem for marking. For the most part, we want to +pretend that the expression of an autoclosure is being evaluated in the +enclosing context; we don't want to have to mark both a call within the +autoclosure and the call to the function taking the autoclosure! We +should teach the type-checking pass to recognize this pattern: a call to +a function that `throwsIf` an autoclosure argument does. + +There's a similar problem with functions that are supposed to feel like +statements. We want you to be able to write: + + autoreleasepool { + let string = parseString(try) + ... + } + +without marking the call to `autoreleasepool`, because this undermines +the ability to write functions that feel like statements. However, there +are other important differences between these trailing-closure uses and +true built-in statements, such as the behavior of `return`, `break`, and +`continue`. An attribute which marks the function as being +statement-like would be a necessary step towards addressing both +problems. Doing this reliably in closures would be challenging, however. + +#### Asserting markers + +Typed propagation is a hypothesis-checking mechanism and so suffers from +the standard problem of false positives. (Basic soundness eliminates +false negatives, of course: the compiler is supposed to force +programmers to deal with *every* source of error.) In this case, a false +positive means a situation where an API is declared to throw but an +error is actually dynamically impossible. + +For example, a function to load an image from a URL would usually be +designed to produce an error if the image didn't exist, the connection +failed, the file data was malformed, or any of a hundred other problems +arose. The programmer should be expected to deal with that error in +general. But a programmer might reasonably use the same API to load an +image completely under their control, e.g. from their program's private +resources. We shouldn't make it too syntactically inconvenient to "turn +off" error-checking for such calls. + +One important point is that we don't want to make it too easy to +*ignore* errors. Ignored errors usually lead to a terrible debugging +experience, even if the error is logged with a meaningful stack trace; +the full context of the failure is lost and can be difficult to +reproduce. Ignored errors also have a way of compounding, where an error +that's "harmlessly" ignored at one layer of abstraction causes another +error elsewhere; and of course the second error can be ignored, etc., +but only by making the program harder and harder to understand and +debug, leaving behind log files that are increasingly jammed with the +detritus of a hundred ignored errors. And finally, ignoring errors +creates a number of type-safety and security problems by encouraging +programs to blunder onwards with meaningless data and broken invariants. + +Instead, we just want to make it (comparatively) easy to turn a static +problem into a dynamic one, much as assertions and the ! operator do. Of +course, this needs to be an explicit operation, because otherwise we +would completely lose typed propagation; and it should be call-specific, +so that the programmer has to make an informed decision about individual +operations. But we already have an explicit, call-site-specific +annotation: the `try` operator. So the obvious solution is to allow a +variant of `try` that asserts that an error is not thrown out of its +operand; and the obvious choice there within our existing design +language is to use the universal "be careful, this is unsafe" marker by +making the keyword `try!`. + +It's reasonable to ask whether `try!` is actually *too* easy to write, +given that this is, after all, an unsafe operation. One quick rejoinder +is that it's no worse than the ordinary `!` operator in that sense. Like +`!`, it's something that a cautious programmer might want to investigate +closer, and you can easily imagine codebases that expect uses of it to +always be explained in comments. But more importantly, just like `!` +it's only *statically* unsafe, and it will reliably fail when the +programmer is wrong. Therefore, while you can easily imagine (and +demonstrate) uncautious programmers flailing around with it to appease +the type-checker, that's not actually a tenable position for the overall +program: eventually the programmer will have to learn how to use the +feature, or else their program simply won't run. + +Furthermore, while `try!` does somewhat undermine error-safety in the +hands of a careless programmer, it's still better to promote this kind +of unsafety than to implicitly promote the alternative. A careless +programmer isn't going to write good error handling just because we +don't give them this feature. Instead, they'll write out a `do/catch` +block, and the natural pressure there will be to silently swallow the +error --- after all, that takes less boilerplate than asserting or +logging. + +In a future release, when we add support for universal errors, we'll +need to reconsider the behavior of `try!`. One possibility is that +`try!` should simply start propagating its operand as a universal error; +this would allow emergency recovery. Alternatively, we may want `try!` +to assert that even universal errors aren't thrown out of it; this would +provide a more consistent language model between the two kinds of +errors. But we don't need to think too hard about this yet. + +### Other syntax + +#### Clean-up actions + +Swift should provide a statement for cleaning up with an *ad hoc* +action. + +Overall, I think it is better to use a Go-style `defer` than a +Java-style `try ... finally`. While this makes the exact order of +execution more obscure, it does make it obvious that the clean-up *will* +be executed without any further analysis, which is something that +readers will usually be interested in. + +Unlike Go, I think this should be tied to scope-exit, not to +function-exit. This makes it very easy to know the set of `defer` +actions that will be executed when a scope exits: it's all the `defer` +statement in exactly that scope. In contrast, in Go you have to +understand the dynamic history of the function's execution. This also +eliminates some semantic and performance oddities relating to variable +capture, since the `defer` action occurs with everything still in scope. +One downside is that it's not as good for "transactional" idioms which +push an undo action for everything they do, but that style has +composition problems across function boundaries anyway. + +I think `defer` is a reasonable name for this, although we might also +consider `finally`. I'll use `defer` in the rest of this proposal. + +`defer` may be followed by an arbitrary statement. The compiler should +reject an action that might terminate early, whether by throwing or with +`return`, `break`, or `continue`. + +Examples: + + if exists(filename) { + let file = open(filename, O_READ) + defer close(file) + + while let line = try file.readline() { + ... + } + + // close occurs here, at the end of the formal scope. + } + +We should consider providing a convenient way to mark that a `defer` +action should only be taken if an error is thrown. This is a convenient +shorthand for controlling the action with a flag that's only set to true +at the end of an operation. The flag approach is often more useful, +since it allows the action to be taken for *any* early exit, e.g. a +`return`, not just for error propagation. + +#### `using` + +Swift should consider providing a `using` statement which acquires a +resource, holds it for a fixed period of time, optionally binds it to a +name, and then releases it whenever the controlled statement exits. + +`using` has many similarities to `defer`. It does not subsume `defer`, +which is useful for many ad-hoc and tokenless clean-ups. But it is +convenient for the common pattern of a type-directed clean-up. + +We do not expect this feature to be necessary in the first release. + +### C and Objective-C Interoperation + +It's of paramount importance that Swift's error model interact as +cleanly with Objective-C APIs as we can make it. + +In general, we want to try to import APIs that produce errors as +throwing; if this fails, we'll import the API as an ordinary +non-throwing function. This is a safe approach only under the assumption +that importing the function as throwing will require significant changes +to the call. That is, if a developer writes code assuming that an API +will be imported as throwing, but in fact Swift fails to import the API +that way, it's important that the code doesn't compile. + +Fortunately, this is true for the common pattern of an error +out-parameter: if Swift cannot import the function as throwing, it will +leave the out-parameter in place, and the compiler will complain if the +developer fails to pass an error argument. However, it is possible to +imagine APIs where the "meat" of the error is returned in a different +way; consider a POSIX API that simply sets `errno`. Great care would +need to be taken when such an API is only partially imported as +throwing. + +Let's wade into the details. + +#### Error types + +`NSError` and `CFError` should implement the `ErrorType` protocol. It +should be possible to turn an arbitrary Swift `enum` that conforms to +`ErrorType` into an `NSError` by using the qualified type name as the +domain key, the enumerator as the error code, and turning the payload +into user data. + +Recognizing system enums as error domains is a matter of annotation. +Most likely, Swift will just special-case a few common domains in the +first release. + +#### Objective-C method error patterns + +The most common error pattern in ObjC by far is for a method to have an +autoreleased `NSError**` out-parameter. We don't currently propose +automatically importing anything as `throws` when it lacks such a +parameter. + +If any APIs take an `NSError**` and *don't* intend for it to be an error +out-parameter, they will almost certainly need it to be marked. + +##### Detecting an error + +Many of these methods have some sort of significant result which is used +for testing whether an error occurred: + +- The most common pattern is a `BOOL` result, where a false value + means an error occurred. This seems unambiguous. + + Swift should import these methods as if they'd returned `Void`. + +- Also common is a pointer result, where a `nil` result usually means + an error occurred. + + I've been told that there are some exceptions to this rule, where a + `nil` result is valid and the caller is apparently meant to check + for a non-`nil` error. I haven't been able to find any such APIs in + Cocoa, though; the claimed APIs I've been referred to do have + nullable results, but returned via out-parameters with a BOOL + formal result. So it seems to be a sound policy decision for + Objective-C that `nil` results are errors by default. CF might be a + different story, though. + + When a `nil` result implies that an error has occurred, Swift should + import the method as returning a non-optional result. + +- A few CF APIs return `void`. As far as I can tell, for all of these, + the caller is expected to check for a non-`nil` error. + +For other sentinel cases, we can consider adding a new clang attribute +to indicate to the compiler what the sentinel is: + +- There are several APIs returning `NSInteger` or `NSUInteger`. At + least some of these return 0 on error, but that doesn't seem like a + reasonable general assumption. +- `AVFoundation` provides a couple methods returning + `AVKeyValueStatus`. These produce an error if the API returned + `AVKeyValueStatusFailed`, which, interestingly enough, is not the + zero value. + +The clang attribute would specify how to test the return value for an +error. For example: + + + (NSInteger)writePropertyList:(id)plist + toStream:(NSOutputStream *)stream + format:(NSPropertyListFormat)format + options:(NSPropertyListWriteOptions)opt + error:(out NSError **)error + NS_ERROR_RESULT(0) + + - (AVKeyValueStatus)statusOfValueForKey:(NSString *)key + error:(NSError **) + NS_ERROR_RESULT(AVKeyValueStatusFailed); + +We should also provide a Clang attribute which specifies that the +correct way to test for an error is to check the out-parameter. Both of +these attributes could potentially be used by the static analyzer, not +just Swift. (For example, they could try to detect an invalid error +check.) + +A constant value would be sufficient for the cases I've seen, but if the +argument has to generalized to a simple expression, that's still +feasible. + +##### The error parameter + +The obvious import rule for Objective-C methods with `NSError**` +out-parameters is to simply mark them `throws` and remove the selector +clause corresponding to the out-parameter. That is, a method like this +one from `NSAttributedString`: + + - (NSData *)dataFromRange:(NSRange)range + documentAttributes:(NSDictionary *)dict + error:(NSError **)error; + +would be imported as: + + func dataFromRange(range: NSRange, + documentAttributes dict: NSDictionary) throws -> NSData + +However, applying this rule haphazardly causes problems for Objective-C +interoperation, because multiple methods can be imported the same way. +The model is far more comprehensible to both compiler and programmer if +the original Objective-C declaration can be unambiguously reconstructed +from a Swift declaration. + +There are two sources of this ambiguity: + +- The error parameter could have appeared at an arbitrary position in + the selector; that is, both `foo:bar:error:` and `foo:error:bar:` + would appear as `foo:bar:` after import. +- The error parameter could have had an arbitrary selector chunk; that + is, both `foo:error:` and `foo:withError:` would appear as `foo:` + after import. + +To allow reconstruction, then, we should only apply the rule when the +error parameter is the last parameter and the corresponding selector is +either `error:` or the first chunk. Empirically, this seems to do the +right thing for all but two sets of APIs in the public API: + +- The `ISyncSessionDriverDelegate` category on `NSObject` declares + half-a-dozen methods like this: + + - (BOOL)sessionDriver:(ISyncSessionDriver *)sender + didRegisterClientAndReturnError:(NSError **)outError; + + Fortunately, these delegate methods were all deprecated in Lion, and + Swift currently doesn't even import deprecated methods. + +- `NSFileCoordinator` has half a dozen methods where the `error:` + clause is second-to-last, followed by a block argument. These + methods are not deprecated as far as I know. + +Of course, user code could also fail to follow this rule. + +I think it's acceptable for Swift to just not import these methods as +`throws`, leaving the original error parameter in place exactly as if +they didn't follow an intelligible pattern in the header. + +This translation rule would import methods like this one from +`NSDocument`: + + - (NSDocument *)duplicateAndReturnError:(NSError **)outError; + +like so: + + func duplicateAndReturnError() throws -> NSDocument + +Leaving the `AndReturnError` bit around feels unfortunate to me, but I +don't see what we could do without losing the ability to automatically +reconstruct the Objective-C signature. This pattern is common but hardly +universal; consider this method from `NSManagedObject`: + + - (BOOL)validateForDelete:(NSError **)error; + +This would be imported as: + + func validateForDelete() throws + +This seems like a really nice import. + +#### CoreFoundation functions + +CF APIs use `CFErrorRef` pretty reliably, but there are two problems. + +First, we're not as confident about the memory management rules for the +error object. Is it always returned at +1? + +Second, I'm not as confident about how to detect that an error has +occurred: + +- There are a lot of functions that return `Boolean` or `bool`. It's + likely that these functions consistently use the same convention as + Objective-C: false means error. +- Similarly, there are many functions that return an object reference. + Again, we'd need a policy on whether to treat `nil` results + as errors. +- There are a handful of APIs that return a `CFIndex`, all with + apparently the same rule that a zero value means an error. (These + are serialization APIs, so writing nothing seems like a + reasonable error.) But just like Objective-C, that does not seem + like a reasonable default assumption. +- `ColorSyncProfile` has several related functions that return + `float`! These are both apparently meant to be checked by testing + whether the error result was filled in. + +There are also some APIs that do not use `CFErrorRef`. For example, most +of the `CVDisplayLink` APIs in CoreVideo returns their own `CVReturn` +enumeration, many with more than one error value. Obviously, these will +not be imported as throwing unless CoreVideo writes an overlay. + +#### Other C APIs + +In principle, we could import POSIX functions into Swift as throwing +functions, filling in the error from `errno`. It's nearly impossible to +imagine doing this with an automatic import rule, however; much more +likely, we'd need to wrap them all in an overlay. + +### Implementation design + +Error propagation for the kinds of explicit, typed errors that I've been +focusing on should be handled by implicit manual propagation. It would +be good to bias the implementation somewhat towards the non-error path, +perhaps by moving error paths to the ends of functions and so on, and +perhaps even by processing cleanups with an interpretive approach +instead of directly inlining that code, but we should not bias so +heavily as to seriously compromise performance. In other words, we +should not use table-based unwinding. + +Error propagation for universal errors should be handled by table-based +unwinding. `catch` handlers can catch both, mapping unwind exceptions to +`ErrorType` values as necessary. With a carefully-designed +interpretation function aimed to solve the specific needs of Swift, we +can avoid most of the code-size impact by shifting it to the unwind +tables, which needn't ever be loaded in the common case. diff --git a/docs/ErrorHandlingRationale.rst b/docs/ErrorHandlingRationale.rst deleted file mode 100644 index a6496801d6733..0000000000000 --- a/docs/ErrorHandlingRationale.rst +++ /dev/null @@ -1,2035 +0,0 @@ -Error Handling Rationale and Proposal -************************************* - -This paper surveys the error-handling world, analyzes various ideas -which have been proposed or are in practice in other languages, and -ultimately proposes an error-handling scheme for Swift together -with import rules for our APIs. - -Fundamentals -============ - -I need to establish some terminology first. - -Kinds of propagation --------------------- - -I've heard people talk about **explicit vs. implicit propagation**. -I'm not going to use those terms, because they're not helpful: there -are at least three different things about error-handling that can be -more or less explicit, and some of the other dimensions are equally -important. - -The most important dimensions of variation are: - -* Whether the language allows functions to be designated as producing - errors or not; such a language has **typed propagation**. - -* Whether, in a language with typed propagation, the default rule is - that that a function can produce an error or that it can't; this - is the language's **default propagation rule**. - -* Whether, in a language with typed propagation, the language enforces - this statically, so that a function which cannot produce an error - cannot call a function which can without handling it; such a - language has **statically-enforced typed propagation**. (A language - could instead enforce this dynamically by automatically inserting - code to assert if an error propagates out. C++ does this.) - -* Whether the language requires all potential error sites to be - identifiable as potential error sites; such a language has **marked - propagation**. - -* Whether propagation is done explicitly with the normal data-flow and - control-flow tools of the language; such a language has **manual - propagation**. In contrast, a language where control implicitly - jumps from the original error site to the proper handler has - **automatic propagation**. - -Kinds of error --------------- - -What is an error? There may be many different possible error -conditions in a program, but they can be categorized into several -kinds based on how programmers should be expected to react to them. -Since the programmer is expected to react differently, and since the -language is the tool of the programmer's reaction, it makes sense for -each group to be treated differently in the language. - -To be clear, in many cases the kind of error reflects a conscious -decision by the author of the error-producing code, and different -choices might be useful in different contexts. The example I'm going -to use of a "simple domain error" could easily be instead treated as a -"recoverable error" (if the author expected automatic propagation to -be more useful than immediate recovery) or even a "logic failure" (if -the author wished to prevent speculative use, e.g. if checking the -precondition was very expensive). - -In order of increasing severity and complexity: - -Simple domain errors -~~~~~~~~~~~~~~~~~~~~ - -A simple domain error is something like calling ``String.toInt()`` on a -string that isn't an integer. The operation has an obvious -precondition about its arguments, but it's useful to be able to pass -other values to test whether they're okay. The client will often -handle the error immediately. - -Conditions like this are best modeled with an optional return value. -They don't benefit from a more complex error-handling model, and using -one would make common code unnecessarily awkward. For example, -speculatively trying to parse a ``String`` as an integer in Java -requires catching an exception, which is far more syntactically -heavyweight (and inefficient without optimization). - -Because Swift already has good support for optionals, these conditions -do not need to be a focus of this proposal. - -Recoverable errors -~~~~~~~~~~~~~~~~~~ - -Recoverable errors include file-not-found, network timeouts, and -similar conditions. The operation has a variety of possible error -conditions. The client should be encouraged to recognize the -possibility of the error condition and consider the right way to -handle it. Often, this will be by aborting the current operation, -cleaning up after itself if needed, and propagating the error to a -point where it can more sensibly be handled, e.g. by reporting it to -the user. - -These are the error conditions that most of our APIs use ``NSError`` and -``CFError`` for today. Most libraries have some similar notion. This -is the focus of this proposal. - -Universal errors -~~~~~~~~~~~~~~~~ - -The difference between universal errors and ordinary recoverable -errors is less the kind of error condition and more the potential -sources of the error in the language. An error is universal if it -could arise from such a wealth of different circumstances that it -becomes nearly impracticable for the programmer to directly deal with -all the sources of the error. - -Some conditions, if they are to be brought into the scope of -error-handling, can only conceivably be dealt with as universal -errors. These include: - -* Asynchronous conditions, like the process receiving a ``SIGINT``, or - the current thread being cancelled by another thread. These - conditions could, in principle, be delivered at an arbitrary - instruction boundary, and handling them appropriately requires - extraordinary care from the programmer, the compiler, and the - runtime. - -* Ubiquitous errors, like running out of memory or overflowing the - stack, that essentially any operation can be assumed to potentially - do. - -But other kinds of error condition can essentially become universal -errors with the introduction of abstraction. Reading the size of a -collection, or reading a property of an object, is not an operation -that a programmer would normally expect to produce an error. However, -if the collection is actually backed by a database, the database query -might fail. If the user must write code as if any opaque abstraction -might produce an error, they are stuck in the world of universal -errors. - -Universal errors mandate certain language approaches. Typed -propagation of universal errors is impossible, other than special -cases which can guarantee to not produce errors. Marked propagation -would provoke a user revolt. Propagation must be automatic, and the -implementation must be "zero-cost", or as near to it as possible, -because checking for an error after every single operation would be -prohibitive. - -For these reasons, in our APIs, universal error conditions are usually -implemented using Objective-C exceptions, although not all uses of -Objective-C exceptions fall in this category. - -This combination of requirements means that all operations must be -implicitly "unwindable" starting from almost any call site it makes. -For the stability of the system, this unwinding process must restore -any invariants that might have been temporarily violated; but the -compiler cannot assist the programmer in this. The programmer must -consciously recognize that an error is possible while an invariant is -broken, and they must do this proactively --- that, or track it down -when they inevitably forget. This requires thinking quite rigorously -about one's code, both to foresee all the error sites and to recognize -that an important invariant is in flux. - -How much of a problem this poses depends quite a lot on the code being -written. There are some styles of programming that make it pretty -innocuous. For example, a highly functional program which -conscientiously kept mutation and side-effects to its outermost loops -would naturally have very few points where any invariants were in -flux; propagating an error out of an arbitrary place within an -operation would simply abandon all the work done up to that point. -However, this happy state falls apart quite quickly the more that -mutation and other side-effects come into play. Complex mutations -cannot be trivially reversed. Packets cannot be unsent. And it would -be quite amazing for us to assert that code shouldn't be written that -way, understanding nothing else about it. As long as programmers do -face these issues, the language has some responsibility to help them. - -Therefore, in my judgment, promoting the use of universal errors is -highly problematic. They undermine the easy comprehension of code, -and they undermine the language's ability to help the programmer -reason about errors. This design will instead focus on explicitly -trackable errors of the sort that ``NSError`` is used for today on Apple -platforms. - -However, there are some important reasons not to rule out universal -errors completely: - -* They remain the only viable means of bringing certain error - conditions into the error-handling model, as discussed above. Of - these, most run into various objections; the most important - remaining use case is "escaping", where an unexpected implementation - of an API that was not designed to throw finds itself needing to. - -* Objective-C and C++ exceptions are a legitimate interoperation - problem on any conceivable platform Swift targets. Swift must have - some sort of long-term answer for them. - -These reasons don't override the problems with universal errors. It -is inherently dangerous to implicitly volunteer functions for -unwinding from an arbitrary point. We don't want to promote this -model. However, it is certainly possible to write code that handles -universal errors correctly; and pragmatically, unwinding through most -code will generally just work. Swift could support a secondary, -untyped propagation mechanism using "zero-cost" exceptions. Code can -be written carefully to minimize the extent of implicit unwinding, -e.g. by catching universal errors immediately after calling an -"escaping" API and rethrowing them with normal typed propagation. - -However, this work is outside of the scope of Swift 2.0. We can -comfortably make this decision because doing so doesn't lock us out of -implementing it in the future: - -- We do not currently support propagating exceptions through Swift - functions, so changing ``catch`` to catch them as well would not be - a major compatibility break. - -- With some admitted awkwardness, external exceptions can be reflected - into an ``ErrorType`` - like model automatically by the catch - mechanism. - -- In the meanwhile, developers who must handle an Objective-C - exception can always do so by writing a stub in Objective-C to - explicitly "bridge" the exception into an ``NSError`` out parameter. - This isn't ideal, but it's acceptable. - -Logic failures -~~~~~~~~~~~~~~ - -The final category is logic failures, including out of bounds array -accesses, forced unwrap of ``nil`` optionals, and other kinds of -assertions. The programmer has made a mistake, and the failure should -be handled by fixing the code, not by attempting to recover -dynamically. - -High-reliability systems may need some way to limp on even after an -assertion failure. Tearing down the process can be viewed as a vector -for a denial-of-service attack. However, an assertion failure might -indicate that the process has been corrupted and is under attack, and -limping on anyway may open the system up for other, more serious forms -of security breach. - -The correct handling of these error conditions is an open question and -is not a focus of this proposal. Should we decide to make them -recoverable, they will likely follow the same implementation mechanism -as universal errors, if not necessarily the same language rules. - -Analysis -======== - -Let's take a deeper look into the different dimensions of -error-handling I laid out above. - -Propagation methods -------------------- - -At a language level, there are two basic ways an error can be -propagated from an error site to something handling it. - -The first is that it can be done with the normal evaluation, data -flow, and control flow processes of the language; let's call this -**manual propagation**. Here's a good example of manual propagation -using special return values in an imperative language, C: - -.. code-block:: c - - struct object *read_object(void) { - char buffer[1024]; - ssize_t numRead = read(0, buffer, sizeof(buffer)); - if (numRead < 0) return NULL; - ... - } - -Here's an example of manual propagation of an error value through -out-parameters in another imperative language, Objective-C: - -.. code-block:: objc - - - (BOOL) readKeys: (NSArray**) strings error: (NSError**) err { - while (1) { - NSString *key; - if ([self readKey: &key error: err]) { - return TRUE; - } - ... - } - ... - } - -Here's an example of manual propagation using an ADT in an impure -functional language, SML; it's somewhat artificial because the SML -library actually uses exceptions for this: - -.. code-block:: sml - - fun read_next_cmd () = - case readline(stdin) of - NONE => NONE - | SOME line => if ... - -All of these excerpts explicitly test for errors using the language's -standard tools for data flow and then explicitly bypass the evaluation -of the remainder of the function using the language's standard tools -for control flow. - -The other basic way to propagate errors is in some hidden, more -intrinsic way not directly reflected in the ordinary control flow -rules; let's call this **automatic propagation**. Here's a good -example of automatic propagation using exceptions in an imperative -language, Java: - -.. code-block:: java - - String next = readline(); - -If ``readline`` encounters an error, it throws an exception; the -language then terminates scopes until it dynamically reaches a ``try`` -statement with a matching handler. Note the lack of any code at all -implying that this might be happening. - -The chief disadvantages of manual propagation are that it's tedious to -write and requires a lot of repetitive boilerplate. This might sound -superficial, but these are serious concerns. Tedium distracts -programmers and makes them careless; careless error-handling code can -be worse than useless. Repetitive boilerplate makes code less -readable, hurting maintainability; it occupies the programmer's time, -creating opportunity costs; it discourages handling errors *well* by -making it burdensome to handle them *at all*; and it encourages -shortcuts (such as extensive macro use) which may undermine other -advantages and goals. - -The chief disadvantage of automatic propagation is that it obscures -the control flow of the code. I'll talk about this more in the next -section. - -Note that automatic propagation needn't be intrinsic in a language. -The propagation is automatic if it doesn't correspond to visible -constructs in the source. This effect can be duplicated as a library -with any language facility that allows restructuring of code -(e.g. with macros or other term-rewriting facilities) or overloading -of basic syntax (e.g. Haskell mapping its ``do`` notation onto monads). - -Note also that multiple propagation strategies may be "in play" for -any particular program. For example, Java generally uses exceptions -in its standard libraries, but some specific APIs might opt to instead -return ``null`` on error for efficiency reasons. Objective-C provides a -fairly full-featured exceptions model, but the standard APIs (with a -few important exceptions) reserve them solely for unrecoverable -errors, preferring manual propagation with ``NSError`` out-parameters -instead. Haskell has a large number of core library functions which -return ``Maybe`` values to indicate success or error, but it also offers -at least two features resembling traditional, -automatically-propagating exceptions (the ``ErrorT`` monad transform and -exceptions in the ``IO`` monad). - -So, while I'm going to talk as if languages implement a single -propagation strategy, it should be understood that reality will always -be more complex. It is literally impossible to prevent programmers -from using manual propagation if they want to. Part of the proposal -will discuss using multiple strategies at once. - -Marked propagation ------------------- - -Closely related to the question of whether propagation is manual or -automatic is whether it is marked or unmarked. Let's say that a -language uses **marked propagation** if there is something *at the -call site* which indicates that propagation is possible from that -point. - -To a certain extent, every language using manual propagation uses -marked propagation, since the manual code to propagate the error -approximately marks the call which generated the error. However, it -is possible for the propagation logic to get separated from the call. - -Marked propagation is at odds with one other major axis of language -design: a language can't solely use marked propagation if it ever -performs implicit operations that can produce errors. For example, a -language that wanted out-of-memory conditions to be recoverable errors -would have to consider everything that could allocate memory to a -source of propagation; in a high-level language, that would include a -large number of implicit operations. Such a language could not claim -to use marked propagation. - -The reason this all matters is because unmarked propagation is a -pretty nasty thing to end up with; it makes it impossible to directly -see what operations can produce errors, and therefore to directly -understand the control flow of a function. This leaves you with two -options as a programmer: - -- You can carefully consider the actual dynamic behavior of every - function called by your function. - -- You can carefully arrange your function so that there are no - critical sections where an universal error can leave things in an - unwanted state. - -There are techniques for making the second more palatable. Chiefly, -they involve never writing code that relies on normal control flow to -maintain invariants and clean up after an operation; for example, -always using constructors and destructors in C++ to manage resources. -This is compulsory in C++ with exceptions enabled because of the -possibility of implicit code that can throw, but it could -theoretically be used in other languages. However, it still requires -a fairly careful and rigorous style of programming. - -It is possible to imagine a marked form of automatic propagation, -where the propagation itself is implicit except that (local) -origination points have to be explicitly marked. This is part of our -proposal, and I'll discuss it below. - - -Typed propagation ------------------ - -The next major question is whether error propagation is explicitly -tracked and limited by the language. That is, is there something -explicitly *in the declaration of a function* that tells the -programmer whether it can produce errors? Let's call this **typed -propagation**. - - -Typed manual propagation -~~~~~~~~~~~~~~~~~~~~~~~~ - -Whether propagation is typed is somewhat orthogonal to whether it's -manual or marked, but there are some common patterns. The most -dominant forms of manual propagation are all typed, since they pass -the failure out of the callee, either as a direct result or in an -out-parameter. - -Here's another example of an out-parameter: - -.. code-block:: objc - - - (instancetype)initWithContentsOfURL:(NSURL *)url encoding:(NSStringEncoding)enc error:(NSError **)error; - -Out-parameters have some nice advantages. First, they're a reliable -source of marking; even if the actual propagation gets separated from -the call, you can always detect a call that can generate errors as -long as its out-parameter has a recognizable name. Second, some of -the boilerplate can be shared, because you can use the same variable -as an out-parameter multiple times; unfortunately, you can't use this -to "cheat" and only check for an error once unless you have some -conventional guarantee that later calls won't spuriously overwrite the -variable. - -A common alternative in functional languages is to return an ``Either`` -type:: - - trait Writer { - fn write_line(&mut self, s: &str) -> Result<(), IoError>; - } - -This forces the caller to deal with the error if they want to use the -result. This works well unless the call does not really have a -meaningful result (as ``write_line`` does not); then it depends on -whether language makes it easy to accidentally ignore results. It -also tends to create a lot of awkward nesting:: - - fn parse_two_ints_and_add_them() { - match parse_int() { - Err e => Err e - Ok x => match parse_int() { - Err e => Err e - Ok y => Ok (x + y) - } - } - } - -Here, another level of nesting is required for every sequential -computation that can fail. Overloaded evaluation syntax like -Haskell's ``do`` notation would help with both of these problems, but -only by switching to a kind of automatic propagation. - -Manual propagation can be untyped if it occurs through a side channel. -For example, consider an object which set a flag on itself when it -encountered an error instead of directly returning it; or consider a -variant of POSIX which expected you to separately check ``errno`` to see -if a particular system call failed. - - -Typed automatic propagation -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Languages with typed automatic propagation vary along several -dimensions. - -The default typing rule -^^^^^^^^^^^^^^^^^^^^^^^ - -The most important question is whether you opt in to producing errors -or opt out of them. That is, is a function with no specific -annotation able to produce errors or not? - -The normal resilience guideline is that you want the lazier option to -preserve more flexibility for the implementation. A function that can -produce errors is definitely more flexible, since it can do more -things. Contrariwise, changing a function that doesn't produce errors -into a function that does clearly changes its contract in ways that -callers need to respond to. Unfortunately, this has some unpleasant -consequences: - -- Marked propagation would become very burdensome. Every call would - involve an annotation, either on the function (to say it cannot - generate errors) or on the call site (to mark propagation). Users - would likely rebel against this much bookkeeping. - -- Most functions cannot generate recoverable errors in the way I've - defined that. That is, ignoring sources of universal errors, most - functions can be reasonably expected to not be able to produce - errors. But if that's not the default state, that means that most - functions would need annotations; again, that's a lot of tedious - bookkeeping. It's also a lot of clutter in the API. - -- Suppose that you notice that a function incorrectly lacks an - annotation. You go to fix it, but you can't without annotating all - of the functions it calls, ad infinitum; like ``const`` correctness in - C++, the net effect is to punish conscientious users for trying to - improve their code. - -- A model which pretends that every function is a source of errors is - likely to be overwhelming for humans. Programmers ought to think - rigorously about their code, but expecting them to also make - rigorous decisions about all the code their code touches is probably - too much. Worse, without marked propagation, the compiler can't - really help the programmer concentrate on the known-possible sources - of error. - -- The compiler's analysis for code generation has to assume that all - sorts of things can produce errors when they really can't. This - creates a lot of implicit propagation paths that are actually 100% - dead, which imposes a serious code-size penalty. - -The alternative is to say that, by default, functions are not being -able to generate errors. This agrees with what I'm assuming is the -most common case. In terms of resilience, it means expecting users to -think more carefully about which functions can generate errors before -publishing an API; but this is similar to how Swift already asks them -to think carefully about types. Also, they'll have at least added the -right set of annotations for their initial implementation. So I -believe this is a reasonable alternative. - -Enforcement -^^^^^^^^^^^ - -The next question is how to enforce the typing rules that prohibit -automatic propagation. Should it be done statically or dynamically? -That is, if a function claims to not generate errors, and it calls a -function that generates errors without handling the error, should that -be a compiler error or a runtime assertion? - -The only real benefit of dynamic enforcement is that it makes it -easier to use a function that's incorrectly marked as being able to -produce errors. That's a real problem if all functions are assumed to -produce errors by default, because the mistake could just be an error -of omission. If, however, functions are assumed to not produce -errors, then someone must have taken deliberate action that introduced -the mistake. I feel like the vastly improved static type-checking is -worth some annoyance in this case. - -Meanwhile, dynamic enforcement undermines most of the benefits of -typed propagation so completely that it's hardly worth considering. -The only benefit that really remains is that the annotation serves as -meaningful documentation. So for the rest of this paper, assume that -typed propagation is statically enforced unless otherwise indicated. - -Specificity -^^^^^^^^^^^ - -The last question is how specific the typing should be: should a -function be able to state the specific classes of errors it produces, -or should the annotation be merely boolean? - -Experience with Java suggests that getting over-specific with -exception types doesn't really work out for the best. It's useful to -be able to recognize specific classes of error, but libraries -generally want to reserve flexibility about the exact kind of error -they produce, and so many errors just end up falling into broad -buckets. Different libraries end up with their own library-specific -general error classes, and exceptions list end up just restating the -library's own dependencies or wrapping the underlying errors in ways -that loses critical information. - - -Tradeoffs of typed propagation -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Typed propagation has a number of advantages and disadvantages, mostly -independent of whether the propagation is automatic. - -The chief advantage is that it is safer. It forces programmers to do -*something* to handle or propagate errors. That comes with some -downsides, which I'll talk about, but I see this as a fairly core -static safety guarantee. This is especially important in an -environment where shuttling operations between threads is common, -since it calls out the common situation where an error needs to -propagate back to the originating thread somehow. - -Even if we're settled on using typed propagation, we should be aware -of the disadvantages and investigate ways to ameliorate them: - -- Any sort of polymorphism gets more complicated, especially - higher-order functions. Functions which cannot generate errors are - in principle subtypes of functions which can. But: - - - Composability suffers. A higher-order function must decide - whether its function argument is allowed to generate errors. If - not, the function may be significantly limiting its usability, or - at least making itself much more difficult to use with - error-generating functions. If so, passing a function that does - not may require a conversion (an awkward explicit one if using - manual propagation), and the result of the call will likely claim - to be able to generate errors when, in fact, it cannot. This can - be solved with overloads, but that's a lot of boilerplate and - redundancy, especially for calls that take multiple functions - (like the function composition operator). - - - If an implicit conversion is allowed, it may need to introduce - thunks. In some cases, these thunks would be inlineable --- - except that, actually, it is quite useful for code to be able to - reverse this conversion and dynamically detect functions that - cannot actually generate errors. For example, an algorithm might - be able to avoid some unnecessary bookkeeping if it knows that its - function argument never fails. This poses some representation - challenges. - -- It tends to promote decentralized error handling instead of letting - errors propagate to a level that actually knows how to handle them. - - - Some programmers will always be tempted to incorrectly pepper - their code with handlers that just swallow errors instead of - correctly propagating them to the right place. This is often - worse than useless; it would often be better if the error just - propagated silently, because the result can be a system in an - inconsistent state with no record of why. Good language and - library facilities for propagating errors can help avoid this, - especially when moving actions between threads. - - - There are many situations where errors are not actually possible - because the programmer has carefully restricted the input. For - example, matching :code:``/[0-9]{4}/`` and then parsing the result - as an integer. It needs to be convenient to do this in a context - that cannot actually propagate errors, but the facility to do this - needs to be carefully designed to discourage use for swallowing - real errors. It might be sufficient if the facility does not - actually swallow the error, but instead causes a real failure. - - - It is possible that the ease of higher-order programming in Swift - might ameliorate many of these problems by letting users writing - error-handling combinators. That is, in situations where a lazy - Java programmer would find themselves writing a ``try/catch`` to - swallow an exception, Swift would allow them to do something more - correct with equal convenience. - -One other minor advantage of marked, statically-enforced typed -propagation: it's a boon for certain kinds of refactoring. -Specifically, when a refactor makes an operation error-producing when -it wasn't before, the absence of any those properties makes the -refactor more treacherous and increases the odds of accidentally -introducing a bug. If propagation is untyped, or the typing isn't -statically enforced, the compiler isn't going to help you at all to -find call sites which need to have error-checking code. Even with -static typed propagation, if the propagation isn't marked specifically -on the call site, the compiler won't warn you about calls made from -contexts that can handle or implicitly propagate the error. But if -all these things are true, the compiler will force you to look at all -the existing call sites individually. - - -Error Types ------------ - -There are many kinds of error. It's important to be able to recognize -and respond to specific error causes programmatically. Swift should -support easy pattern-matching for this. - -But I've never really seen a point to coarser-grained categorization -than that; for example, I'm not sure how you're supposed to react to -an arbitrary, unknown IO error. And if there are useful error -categories, they can probably be expressed with predicates instead of -public subclasses. I think we start with a uni-type here and then -challenge people to come up with reasons why they need anything more. - -Implementation design ---------------------- - -There are several different common strategies for implementing -automatic error propagation. (Manual propagation doesn't need special -attention in the implementation design.) - -The implementation has two basic tasks common to most languages: - -* Transferring control through scopes and functions to the appropriate - handler for the error. - -* Performing various semantic "clean up" tasks for the scopes that - were abruptly terminated: - - * tearing down local variables, like C++ variables with - destructors or strong/weak references in ARC-like languages; - - * releasing heap-allocated local variables, like captured variables - in Swift or ``__block`` variables in ObjC; - - * executing scope-specific termination code, like C#'s ``using`` or - Java/ObjC's ``synchronized`` statements; and - - * executing ad hoc cleanup blocks, like ``finally`` blocks in Java - or ``defer`` actions in Swift. - -Any particular call frame on the stack may have clean-ups or potential -handlers or both; call these **interesting frames**. - -Implicit manual propagation -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -One strategy is to implicitly produce code to check for errors and -propagate them up the stack, imitating the code that the programmer -would have written under manual propagation. For example, a function -call could return an optional error in a special result register; the -caller would check this register and, if appropriate, unwind the stack -and return the same value. - -Since propagation and unwinding are explicit in the generated code, -this strategy hurts runtime performance along the non-error path more -than the alternatives, and more code is required to do the explicitly -unwinding. Branches involved in testing for errors are usually very -easy to predict, so in hot code the direct performance impact is quite -small, and the total impact is dominated by decreased code locality. -Code can't always be hot, however. - -These penalties are suffered even by uninteresting frames unless they -appear in tail position. (An actual tail call isn't necessary; there -just can't be anything that error propagation would skip.) And -functions must do some added setup work before returning. - -The upside is that the error path suffers no significant penalties -beyond the code-size impact. The code-size impact can be significant, -however: there is sometimes quite a lot of duplicate code needed for -propagation along the error path. - -This approach is therefore relatively even-handed about the error -vs. the non-error path, although it requires some care in order to -minimize code-size penalties for parallel error paths. - -``setjmp`` / ``longmp`` -~~~~~~~~~~~~~~~~~~~~~~~ - -Another strategy to is to dynamically maintain a thread-local stack of -interesting frames. A function with an interesting frame must save -information about its context in a buffer, like ``setjmp`` would, and -then register that buffer with the runtime. If the scope returns -normally, the buffer is accordingly unregistered. Starting -propagation involves restoring the context for the top of the -interesting-frames stack; the place where execution returns is called -the "landing pad". - -The advantage of this is that uninteresting frames don't need to do -any work; context restoration just skips over them implicitly. This -is faster both for the error and non-error paths. It is also possible -to optimize this strategy so that (unlike ``setjmp``) the test for an -error is implicitly elided: use a slightly different address for the -landing pad, so that propagating errors directly restore to that -location. - -The downside is that saving the context and registering the frame are -not free: - -* Registering the frame requires an access to thread-local state, - which on our platforms means a function call because we're not - willing to commit to anything more specific in the ABI. - -* Jumping across arbitary frames invalidates the callee-save - registers, so the registering frame must save them all eagerly. In - calling conventions with many callee-save registers, this can be - very expensive. However, this is only necessary when it's possible - to resume normal execution from the landing pad: if the landing pad - only has clean-ups and therefore always restarts propagation, those - registers will have been saved and restored further out. - -* Languages like C++, ObjC ARC, and Swift that have non-trivial - clean-ups for many local variables tend to have many functions with - interesting frames. This means both that the context-saving - penalties are higher and that skipping uninteresting frames is a - less valuable optimization. - -* By the same token, functions in those languages often have many - different clean-ups and/or handlers. For example, every new - non-trivial variable might introduce a new clean-up. The function - must either register a new landing pad for each clean-up (very - expensive!) or track its current position in a way that a - function-wide landing pad can figure out what scope it was in. - -This approach can be hybridized with the unwinding approach below so -that the interesting-frames stack abstractly describes the clean-ups -in the frame instead of just restoring control somewhere and expecting -the frame to figure it out. This can decrease the code size impact -significantly for the common case of frames that just need to run some -clean-ups before propagating the error further. It may even -completely eliminate the need for a landing pad. - -The ObjC/C++ exceptions system on iOS/ARM32 is kindof like that -hybrid. Propagation and clean-up code is explicit in the function, -but the registered context includes the "personality" information from -the unwinding tables, which makes the decision whether to land at the -landing pad at all. It also uses an optimized ``setjmp`` implementation -that both avoids some context-saving and threads the branch as -described above. - -The ObjC exceptions system on pre-modern runtimes (e.g. on PPC and -i386) uses the standard ``setjmp`` / ``longjmp`` functions. Every -protected scope saves the context separately. This is all implemented -in a very unsafe way that does not behave well in the presence of -inlining. - -Overall, this approach requires a lot of work in the non-error path -of functions with interesting frames. Given that we expects functions -with interesting frames to be very common in Swift, this is not -an implementation approach we would consider in the abstract. However, -it is the implementation approach for C++/ObjC exceptions on iOS/ARM32, -so we need to at least interoperate with that. - -Table-based unwinding -~~~~~~~~~~~~~~~~~~~~~ - -The final approach is side-table stack unwinding. This relies on -being able to accurately figure out how to unwind through an arbitrary -function on the system, given only the return address of a call it -made and the stack pointer at that point. - -On our system, this proceeds as follows. From an instruction pointer, -the system unwinder looks up what linked image (executable or dylib) -that function was loaded from. The linked image contains a special -section, a table of unwind tables indexed by their offset within the -linked image. Every non-leaf function should have an entry within -this table, which provides sufficient information to unwind the -function from an arbitrary call site. - -This lookup process is quite expensive, especially since it has to -repeat all the way up the stack until something actually handles the -error. This makes the error path extremely slow. However, no -explicit setup code is required along the non-error path, and so this -approach is sometimes known as "zero-cost". That's something of a -misnomer, because it does have several costs that can affect non-error -performance. First, there's a small amount of load-time work required -in order to resolve relocations to symbols used by the unwind tables. -Second, the error path often requires code in the function, which can -decrease code locality even if never executed. Third, the error path -may use information that the non-error path would otherwise discard. -And finally, the unwind tables themselves can be fairly large, -although this is generally only a binary-size concern because they are -carefully arranged to not need to be loaded off of disk unless an -exception is thrown. But overall, "zero-cost" is close enough to -correct. - -To unwind a frame in this sense specifically means: - -* Deciding whether the function handles the error. - -* Cleaning up any interesting scopes that need to be broken down - (either to get to the handler or to leave the function). - -* If the function is being fully unwound, restoring any callee-save - registers which the function might have changed. - -This is language-specific, and so the table contains language-specific -"personality" information, including a reference to a function to -interpret it. This mechanism means that the unwinder is extremely -flexible; not only can it support arbitrary languages, but it can -support different language-specific unwinding table layouts for the -same language. - -Our current personality records for C++ and Objective-C contain just -enough information to decide (1) whether an exception is handled by -the frame and (2) if not, whether a clean-up is currently active. If -either is true, it restores the context of a landing pad, which -manually executes the clean-ups and enters the handler. This approach -generally needs as much code in the function as implicit manual -propagation would. However, we could optimize this for many common -cases by causing clean-ups to be called automatically by the -interpretation function. That is, instead of a landing pad that looks -notionally like this:: - - void *exception = ...; - SomeCXXType::~SomeCXXType(&foo); - objc_release(bar); - objc_release(baz); - _Unwind_Resume(exception); - -The unwind table would have a record that looks notionally like this:: - - CALL_WITH_FRAME_ADDRESS(&SomeCXXType::~SomeCXXType, FRAME_OFFSET_OF(foo)) - CALL_WITH_FRAME_VALUE(&objc_release, FRAME_OFFSET_OF(bar)) - CALL_WITH_FRAME_VALUE(&objc_release, FRAME_OFFSET_OF(baz)) - RESUME - -And no code would actually be needed in the function. This would -generally slow the error path down, because the interpretation -function would have to interpret this mini-language, but it would move -all the overhead out of the function and into the error table, where -it would be more compact. - -This is something that would also benefit C++ code. - -Clean-up actions ----------------- - -Many languages have a built-in language tool for performing arbitrary -clean-up when exiting a scope. This has two benefits. The first is -that, even ignoring error propagation, it acts as a "scope guard" -which ensures that the clean-up is done if the scope is exited early -due to a ``return``, ``break``, or ``continue`` statement; otherwise, the -programmer must carefully duplicate the clean-up in all such places. -The second benefit is that it makes clean-up tractable in the face of -automatic propagation, which creates so many implicit paths of control -flow out of the scope that expecting the programmer to cover them all -with explicit catch-and-rethrow blocks would be ridiculous. - -There's an inherent tension in these language features between putting -explicit clean-up code in the order it will be executed and putting it -near the code it's cleaning up after. The former means that a -top-to-bottom read of the code tells you what actions are being -performed when; you don't have to worry about code implicitly -intervening at the end of a scope. The latter makes it easy to verify -at the point that a clean-up is needed that it will eventually happen; -you don't need to scan down to the finally block and analyze what -happens there. - -``finally`` -~~~~~~~~~~~ - -Java, Objective-C, and many other languages allow ``try`` statements to -take a ``finally`` clause. The clause is an ordinary scope and may take -arbitrary actions. The ``finally`` clause is performed when the -preceding controlled scopes (including any ``catch`` clauses) are exited -in any way: whether by falling off the end, directly branching or -returning out, or throwing an exception. - -``finally`` is a rather awkward and verbose language feature. It -separates the clean-up code from the operation that required it -(although this has benefits, as discussed above). It adds a lot of -braces and indentation, so edits that add new clean-ups can require a -lot of code to be reformatted. When the same scope needs multiple -clean-ups, the programmer must either put them in the same ``finally`` -block (and thus create problems with clean-ups that might terminate -the block early) or stack them up in separate blocks (which can really -obscure the otherwise simple flow of code). - -``defer`` -~~~~~~~~~ - -Go provides a ``defer`` statement that just enqueues arbitrary code to -be executed when the function exits. (More details of this appear in -the survey of Go.) - -This allows the defer action to be written near the code it -"balances", allowing the reader to immediately see that the required -clean-up will be done (but this has drawbacks, as discussed above). -It's very compact, which is nice as most defer actions are short. It -also allow multiple actions to pile up without adding awkward nesting. -However, the function-exit semantics exacerbate the problem of -searching for intervening clean-up actions, and they introduce -semantic and performance problems with capturing the values of local -variables. - -Destructors -~~~~~~~~~~~ - -C++ allows types to define destructor functions, which are called when -a function goes out of scope. - -These are often used directly to clean up the ownership or other -invariants on the type's value. For example, an owning-pointer type -would free its value in its destructor, whereas a hash-table type -would destroy its entries and free its buffer. - -But they are also often used idiomatically just for the implicit -destructor call, as a "scope guard" to ensure that something is done -before the current operation completes. For an example close to my -own heart, a compiler might use such a guard when parsing a local -scope to ensure that new declarations are removed from the scope -chains even if the function exits early due to a parse error. -Unfortunately, since type destructors are C++'s only tool for this -kind of clean-up, introducing ad-hoc clean-up code requires defining a -new type every time. - -The unique advantage of destructors compared to the options above is -that destructors can be tied to temporary values created during the -evaluation of an expression. - -Generally, a clean-up action becomes necessary as the result of some -"acquire" operation that occurs during an expression. ``defer`` and -``finally`` do not take effect until the next statement is reached, -which creates an atomicity problem if code can be injected after the -acquire. (For ``finally``, this assumes that the acquire appears -*before* the ``try``. If instead the acquire appears *within* the -``try``, there must be something which activates the clean-up, and that -has the same atomicity problem.) - -In contrast, if the acquire operation always creates a temporary with -a destructor that does the clean-up, the language automatically -guarantees this atomicity. This pattern is called "resource -acquisition is initialization", or "RAII". Under RAII, all resources -that require clean-up are carefully encapsulated within types with -user-defined destructors, and the act of constructing an object of -that type is exactly the act of acquiring the underlying resource. - -Swift does not support user-defined destructors on value types, but it -does support general RAII-like programming with class types and -``deinit`` methods, although (at the moment) the user must take special -care to keep the object alive, as Swift does not normally guarantee -the destruction order of objects. - -RAII is very convenient when there's a definable "resource" and -somebody's already wrapped its acquisition APIs to return -appropriately-destructed objects. For other tasks, where a reasonable -programmer might balk at defining a new type and possibly wrapping an -API for a single purpose, a more *ad hoc* approach may be warranted. - - -Survey -====== - -C ---- - -C doesn't really have a consensus error-handling scheme. There's a -built-in unwinding mechanism in ``setjmp`` and ``longjmp``, but it's -disliked for a host of good reasons. The dominant idiom in practice -is for a function to encode failure using some unreasonable value for -its result, like a null pointer or a negative count. The bad value(s) -are often function-specific, and sometimes even argument- or -state-specific. - -On the caller side, it is unfortunately idiomatic (in some codebases) -to have a common label for propagating failure at the end of a -function (hence ``goto fail``); this is because there's no inherent -language support for ensuring that necessary cleanup is done before -propagating out of a scope. - -C++ ---- - -C++ has exceptions. Exceptions can have almost any type in the -language. Propagation typing is tied only to declarations; an -indirect function pointer is generally assumed to be able to throw. -Propagation typing used to allow functions to be specific about the -kinds of exceptions they could throw (:code:``throws -(std::exception)``), but this is deprecated in favor of just indicating -whether a function can throw (:code:``noexcept(false)``). - -C++ aspires to making out-of-memory a recoverable condition, and so -allocation can throw. Therefore, it is essentially compulsory for the -language to assume that constructors might throw. Since constructors -are called pervasively and implicitly, it makes sense for the default -rule to be that all functions can throw. Since many error sites are -implicit, there is little choice but to use automatic unmarked -propagation. The only reasonable way to clean up after a scope in -such a world is to allow the compiler to do it automatically. C++ -programmers therefore rely idiomatically on a pattern of shifting all -scope cleanup into the destructors of local variables; sometimes such -local values are created solely to set up a cleanup action in this -way. - -Different error sites occur with a different set of cleanups active, -and there are a large number of such sites. In fact, prior to C++11, -compilers were forced to assume by default that destructor calls could -throw, so cleanups actually created more error sites. This all adds -up to a significant code-size penalty for exceptions, even in projects -which don't directly use them and which have no interest in recovering -from out-of-memory conditions. For this reason, many C++ projects -explicitly disable exceptions and rely on other error propagation -mechanisms, on which there is no widespread consensus. - -Objective C ------------ - -Objective C has a first-class exceptions mechanism which is similar in -feature set to Java's: ``@throw`` / ``@try`` / ``@catch`` / ``@finally``. -Exception values must be instances of an Objective-C class. The -language does a small amount of implicit frame cleanup during -exception propagation: locks held by ``@synchronized`` are released, -stack copies of ``__block`` variables are torn down, and ARC ``__weak`` -variables are destroyed. However, the language does not release -object pointers held in local variables, even (by default) under ARC. - -Objective C exceptions used to be implemented with ``setjmp``, -``longjmp``, and thread-local state managed by a runtime, but the only -surviving platform we support which does that is i386, and all others -now use a "zero-cost" implementation that interoperates with C++ -exceptions. - -Objective C exceptions are *mostly* only used for unrecoverable -conditions, akin to what I called "failures" above. There are a few -major exceptions to this rule, where APIs that do use exceptions to -report errors. - -Instead, Objective C mostly relies on manual propagation, -predominantly using out-parameters of type ``NSError**``. Whether the -call failed is usually *not* indicated by whether a non-``nil`` error -was written into this parameter; calls are permitted both to succeed -and write an error object into the parameter (which should be ignored) -and to report an error without creating an actual error object. -Instead, whether the call failed is reported in the formal return -value. The most common convention is for a false ``BOOL`` result or -null object result to mean an error, but ingenious programmers have -come up with many other conventions, and there do exist APIs where a -null object result is valid. - -CF APIs, meanwhile, have their own magnificent set of somewhat -inconsistent conventions. - -Therefore, we can expect that incrementally improving CF / Objective C -interoperation is going to be a long and remarkably painful process. - - -Java ----- - -Java has a first-class exceptions mechanism with unmarked automatic -propagation: ``throw`` / ``try`` / ``catch`` / ``finally``. Exception values -must be instances of something inheriting from ``Throwable``. -Propagation is generally typed with static enforcement, with the -default being that a call cannot throw exceptions *except* for -subclasses of ``Error`` and ``RuntimeException``. The original intent was -that these classes would be used for catastrophic runtime errors -(``Error``) and programming mistakes caught by the runtime -(``RuntimeException``), both of which we would classify as unrecoverable -failures in our scheme; essentially, Java attempts to promote a fully -statically-enforced model where truly catastrophic problems can still -be handled when necessary. Unfortunately, these motivations don't -seem to have been communicated very well to developers, and the result -is kindof a mess. - -Java allows methods to be very specific about the kinds of exception -they throw. In my experience, exceptions tend to fall into two -categories: - -- There are some very specific exception kinds that callers know to - look for and handle on specific operations. Generally these are - obvious, predictable error conditions, like a host name not - resolving, or like a string not being formatted correctly. - -- There are also a lot of very vague, black-box exception kinds that - can't really be usefully responded to. For example, if a method - throws ``IOException``, there's really nothing a caller can do except - propagate it and abort the current operation. - -So specific typing is useful if you can exhaustively handle a small -number of specific failures. As soon as the exception list includes -any kind of black box type, it might as well be a completely open set. - -C# ---- - -C#'s model is almost exactly like Java's except that it is untyped: -all methods are assumed to be able to throw. For this reason, it also -has a simpler type hierarchy, where all exceptions just inherit from -``Exception``. - -The rest of the hierarchy doesn't really make any sense to me. Many -things inherit directly from ``Exception``, but many other things -inherit from a subclass called ``SystemException``. ``SystemException`` -doesn't seem to be any sort of logical grouping: it includes all the -runtime-assertion exceptions, but it also includes every exception -that's thrown anywhere in the core library, including XML and IO -exceptions. - -C# also has a ``using`` statement, which is useful for binding something -over a precise scope and then automatically disposing it on all paths. -It's just built on top of ``try`` / ``finally``. - -Haskell -------- - -Haskell provides three different common error-propagation mechanisms. - -The first is that, like many other functional languages, it supports -manual propagation with a ``Maybe`` type. A function can return ``None`` -to indicate that it couldn't produce a more useful result. This is -the most common failure method for functions in the functional subset -of the library. - -The ``IO`` monad also provides true exceptions with unmarked automatic -propagation. These exceptions can only be handled as an ``IO`` action, -but are otherwise untyped: there is no way to indicate whether an ``IO`` -action can or cannot throw. Exceptions can be thrown either as an -``IO`` action or as an ordinary lazy functional computation; in the -latter case, the exception is only thrown if the computation is -evaluated for some reason. - -The ``ErrorT`` monad transform provides typed automatic propagation. In -an amusing twist, since the only native computation of ``ErrorT`` is -``throwError``, and the reason to write a computation specifically in -``ErrorT`` is if it's throwing, and every other computation must be -explicitly lifted into the monad, ``ErrorT`` effectively uses marked -propagation by omission, since everything that *can't* throw is -explicitly marked with a ``lift``: - -.. code-block:: haskell - - prettyPrintShiftJIS :: ShiftJISString -> ErrorT TranscodeError IO () - prettyPrintShiftJIS str = do - lift $ putChar '"' -- lift turns an IO computation into an ErrorT computation - case transcodeShiftJISToUTF8 str of - Left error -> throwError error - Right value -> lift $ putEscapedString value - lift $ putChar '"' - -Rust ----- - -Rust distinguishes between *failures* and *panics*. - -A panic is an assertion, designed for what I called logic failures; -there's no way to recover from one, it just immediately crashes. - -A failure is just when a function doesn't produce the value you might -expect, which Rust encourages you to express with either ``Option`` -(for simple cases, like what I described as simple domain errors) or -``Result`` (which is effectively the same, except carrying an error). -In either case, it's typed manual propagation, although Rust does at -least offer a standard macro which wraps the common -pattern-match-and-return pattern for ``Result``. - -The error type in Rust is a very simple protocol, much like this -proposal suggests. - -Go ---- - -Go uses an error result, conventionally returned as the final result -of functions that can fail. The caller is expected to manually check -whether this is nil; thus, Go uses typed manual propagation. - -The error type in Go is an interface named ``error``, with one method -that returns a string description of the error. - -Go has a ``defer`` statement:: - - defer foo(x, y) - -The argument has to be a call (possibly a method call, possibly a call -to a closure that you made specifically to immediately call). All the -operands are evaluated immediately and captured in a deferred action. -Immediately after the function exits (through whatever means), all the -deferred actions are executed in LIFO order. Yes, this is tied to -function exit, not scope exit, so you can have a dynamic number of -deferred actions as a sort of implicit undo stack. Overall, it's a -nice if somewhat quirky way to do ad-hoc cleanup actions. - -It is also a key part of a second, funky kind of error propagation, -which is essentially untyped automatic propagation. If you call -``panic`` --- and certain builtin operations like array accesses behave -like they do --- it immediately unwinds the stack, running deferred -actions as it goes. If a function's deferred action calls ``recover``, -the panic stops, the rest of the deferred actions for the function are -called, and the function returns. A deferred action can write to the -named results, allowing a function to turn a panic error into a -normal, final-result error. It's conventional to not panic over -API boundaries unless you really mean it; recoverable errors are -supposed to be done with out-results. - -Scripting languages -------------------- - -Scripting languages generally all use (untyped, obviously) automatic -exception propagation, probably because it would be quite error-prone -to do manual propagation in an untyped language. They pretty much all -fit into the standard C++/Java/C# style of ``throw`` / ``try`` / ``catch``. -Ruby uses different keywords for it, though. - -I feel like Python uses exceptions a lot more than most other -scripting languages do, though. - -Proposal -======== - -Automatic propagation ---------------------- - -Swift should use automatic propagation of errors, rather than relying -on the programmer to manually check for them and return out. It's -just a lot less boilerplate for common error handling tasks. This -introduces an implicit control flow problem, but we can ameliorate -that with marked propagation; see below. - -There's no compelling reason to deviate from the ``throw`` / ``catch`` -legacy here. There are other options, like ``raise`` / ``handle``. In -theory, switching would somewhat dissociate Swift from the legacy of -exceptions; people coming from other languages have a lot of -assumptions about exceptions which don't necessarily apply to Swift. -However, our error model is similar enough to the standard exception -model that people are inevitably going to make the connection; there's -no getting around the need to explain what we're trying to do. So -using different keywords just seems petty. - -Therefore, Swift should provide a ``throw`` expression. It requires an -operand of type ``Error`` and formally yields an arbitrary type. Its -dynamic behavior is to transfer control to the innermost enclosing -``catch`` clause which is satisfied by the operand. A quick example:: - - if timeElapsed() > timeThreshold { throw HomeworkError.Overworked } - -A ``catch`` clause includes a pattern that matches an error. We want to -repurpose the ``try`` keyword for marked propagation, which it seems to -fit far better, so ``catch`` clauses will instead be attached to a -generalized ``do`` statement:: - - do { - ... - - } catch HomeworkError.Overworked { - // a conditionally-executed catch clause - - } catch _ { - // a catch-all clause - } - -Swift should also provide some tools for doing manual propagation. We -should have a standard Rust-like :code:``Result`` enum in the -library, as well as a rich set of tools, e.g.: - -- A function to evaluate an error-producing closure and capture the - result as a :code:``Result``. - -- A function to unpack a :code:``Result`` by either returning its - value or propagating the error in the current context. - -- A futures library that traffics in :code:``Result`` when - applicable. - -- An overload of ``dispatch_sync`` which takes an error-producing - closure and propagates an error in the current context. - -- etc. - -Typed propagation ------------------ - -Swift should use statically-enforced typed propagation. By default, -functions should not be able to throw. A call to a function which can -throw within a context that is not allowed to throw should be rejected -by the compiler. - -Function types should indicate whether the function throws; this needs -to be tracked even for first-class function values. Functions which -do not throw are subtypes of functions that throw. - -This would be written with a ``throws`` clause on the function -declaration or type:: - - // This function is not permitted to throw. - func foo() -> Int { - // Therefore this is a semantic error. - return try stream.readInt() - } - - // This function is permitted to throw. - func bar() throws -> Int { - return try stream.readInt() - } - - // ‘throws’ is written before the arrow to give a sensible and - // consistent grammar for function types and implicit () result types. - func baz() throws { - if let byte = try stream.getOOB() where byte == PROTO_RESET { - reset() - } - } - - // ‘throws’ appears in a consistent position in function types. - func fred(callback: (UInt8) throws -> ()) throws { - while true { - let code = try stream.readByte() - if code == OPER_CLOSE { return } - try callback(code) - } - } - - // It only applies to the innermost function for curried functions; - // this function has type: - // (Int) -> (Int) throws -> Int - func jerry(i: Int)(j: Int) throws -> Int { - // It’s not an error to use ‘throws’ on a function that can’t throw. - return i + j - } - -The reason to use a keyword here is that it's much nicer for function -declarations, which generally outnumber function types by at least an -order of magnitude. A punctuation mark would be easily lost or -mistaken amidst all the other punctuation in a function declaration, -especially if the punctuation mark were something like ``!`` that can -validly appear at the end of a parameter type. It makes sense for the -keyword to appear close to the return type, as it's essentially a part -of the result and a programmer should be able to see both parts in the -same glance. The keyword appears before the arrow for the simple -reason that the arrow is optional (along with the rest of the return -type) in function and initializer declarations; having the keyword -appear in slightly different places based on the presence of a return -type would be silly and would making adding a non-void return type -feel awkward. The keyword itself should be descriptive, and it's -particularly nice for it to be a form of the verb used by the throwing -expression, conjugated as if performed by the function itself. Thus, -``throw`` becomes ``throws``; if we used ``raise`` instead, this would -be ``raises``, which I personally find unappealing for reasons I'm not -sure I can put a name to. - -It shouldn't be possible to overload functions solely based on whether -the functions throw. That is, this is not legal:: - - func foo() { ... } // called in contexts that cannot throw - func foo() throws { ... } // called in contexts that can throw - -It is valuable to be able to overload higher-order functions based on -whether an argument function throws; it is easy to imagine algorithms -that can be implemented more efficiently if they do not need to worry -about exceptions. (We do not, however, particularly want to encourage -a pattern of duplicating This is straightforward if the primary -type-checking pass is able to reliably decide whether a function value -can throw. - -Typed propagation checking can generally be performed in a secondary -pass over a type-checked function body: if a function is not permitted -to throw, walk its body and verify that there are no ``throw`` -expressions or calls to functions that can ``throw``. If all throwing -calls must be marked, this can be done prior to type-checking to -decide syntactically whether a function can apparently throw; of -course, the later pass is still necessary, but the ability to do this -dramatically simplifies the implementation of the type-checker, as -discussed below. Certain type-system features may need to be -curtailed in order to make this implementation possible for schedule -reasons. (It's important to understand that this is *not* the -motivation for marked propagation. It's just a convenient consequence -that marked propagation makes this implementation possible.) - -Reliably deciding whether a function value can throw is easy for -higher-order uses of declared functions. The problem, as usual, is -anonymous functions. We don't want to require closures to be -explicitly typed as throwing or non-throwing, but the fully-accurate -inference algorithm requires a type-checked function body, and we -can't always type-check an anonymous function independently of its -enclosing context. Therefore, we will rely on being able to do a pass -prior to type-checking to syntactically infer whether a closure -throws, then making a second pass after type-checking to verify the -correctness of that inference. This may break certain kinds of -reasonable code, but the multi-pass approach should let us -heuristically unbreak targeted cases. - -Typed propagation has implications for all kinds of polymorphism: - -Higher-order polymorphism -~~~~~~~~~~~~~~~~~~~~~~~~~ - -We should make it easy to write higher-order functions that behave -polymorphically w.r.t. whether their arguments throw. This can be -done in a fairly simple way: a function can declare that it throws if -any of a set of named arguments do. As an example (using strawman -syntax):: - - func map(array: [T], fn: T throws -> U) throwsIf(fn) -> [U] { - ... - } - -There's no need for a more complex logical operator than disjunction. -You can construct really strange code where a function throws only if -one of its arguments doesn't, but it'd be contrived, and it's hard to -imagine how they could be type-checked without a vastly more -sophisticated approach. Similarly, you can construct situations where -whether a function can throw is value-dependent on some other -argument, like a "should I throw an exception" flag, but it's hard to -imagine such cases being at all important to get right in the -language. This schema is perfectly sufficient to express normal -higher-order stuff. - -In fact, while the strawman syntax above allows the function to be -specific about exactly which argument functions cause the callee to -throw, that's already overkill in the overwhelmingly likely case of a -function that throws if any of its argument functions throw (and -there's probably only one). So it would probably be better to just -have a single ``rethrows`` annotation, with vague plans to allow it -to be parameterized in the future if necessary. - -This sort of propagation-checking would be a straightforward extension -of the general propagation checker. The normal checker sees that a -function isn't allowed to propagate out and looks for propagation -points. The conditional checker sees that a function has a -conditional propagation clause and looks for propagation points, -assuming that the listed functions don't throw (including when looking -at any conditional propagation clauses). The parameter would have to -be a ``let``. - -We probably do need to get higher-order polymorphism right in the -first release, because we will need it for the short-circuiting -operators. - -Generic polymorphism -~~~~~~~~~~~~~~~~~~~~ - -It would be useful to be able to parameterize protocols, and protocol -conformances, on whether the operations produce errors. Lacking this -feature means that protocol authors must decide to either -conservatively allow throwing conformances, and thus force all generic -code using the protocol to deal with probably-spurious errors, or -aggressively forbid them, and thus forbid conformances by types whose -operations naturally throw. - -There are several different ways we could approach this problem, but -after some investigation I feel confident that they're workable. -Unfortunately, they are clearly out-of-scope for the first release. -For now, the standard library should provide protocols that cannot -throw, even though this limits some potential conformances. (It's -worth noting that such conformances generally aren't legal today, -since they'd need to return an error result somehow.) - -A future direction for both generic and higher-order polymorphism is -to consider error propagation to be one of many possible effects in a -general, user-extensible effect tracking system. This would allow the -type system to check that certain specific operations are only allowed -in specific contexts: for example, that a blocking operation is only -allowed in a blocking context. - -Error type -~~~~~~~~~~ - -The Swift standard library will provide ``ErrorType``, a protocol with -a very small interface (which is not described in this proposal). The -standard pattern should be to define the conformance of an ``enum`` to -the type:: - - enum HomeworkError : ErrorType { - case Overworked - case Impossible - case EatenByCat(Cat) - case StopStressingMeWithYourRules - } - -The ``enum`` provides a namespace of errors, a list of possible errors -within that namespace, and optional values to attach to each option. - -For now, the list of errors in a domain will be fixed, but permitting -future extension is just ordinary enum resilience, and the standard -techniques for that will work fine in the future. - -Note that this corresponds very cleanly to the ``NSError`` model of an -error domain, an error code, and optional user data. We expect to -import system error domains as enums that follow this approach and -implement ``ErrorType``. ``NSError`` and ``CFError`` themselves will also -conform to ``ErrorType``. - -The physical representation (still being nailed down) will make it -efficient to embed an ``NSError`` as an ``ErrorType`` and vice-versa. It -should be possible to turn an arbitrary Swift ``enum`` that conforms to -``ErrorType`` into an ``NSError`` by using the qualified type name as the -domain key, the enumerator as the error code, and turning the payload -into user data. - -It's acceptable to allocate memory whenever an error is needed, -but our representation should not inhibit the optimizer from -forwarding a ``throw`` directly to a ``catch`` and removing the -intermediate error object. - -Marked propagation ------------------- - -Swift should use marked propagation: there should be some lightweight -bit of syntax decorating anything that is known be able to throw -(other than a ``throw`` expression itself, of course). - -Our proposed syntax is to repurpose ``try`` as something that can be -wrapped around an arbitrary expression:: - - // This try applies to readBool(). - if try stream.readBool() { - - // This try applies to both of these calls. - let x = try stream.readInt() + stream.readInt() - - // This is a semantic error; it needs a try. - var y = stream.readFloat() - - // This is okay; the try covers the entire statement. - try y += stream.readFloat() - } - -Developers can "scope" the ``try`` very tightly by writing it within -parentheses or on a specific argument or list element:: - - // Semantic error: the try only covers the parenthesized expression. - let x = (try stream.readInt()) + stream.readInt() - - // The try applies to the first array element. Of course, the - // developer could cover the entire array by writing the try outside. - let array = [ try foo(), bar(), baz() ] - -Some developers may wish to do this to make the specific throwing -calls very clear. Other developers may be content with knowing that -something within a statement can throw. - -We also briefly considered the possibility of putting the marker into -the call arguments clause, e.g.:: - - parser.readKeys(&strings, try) - -This works as long as the only throwing calls are written -syntactically as calls; this covers calls to free functions, methods, -and initializers. However, it effectively requires Swift to forbid -operators and property and subscript accessors from throwing, which -may not be a reasonable limitation, especially for operators. It is -also somewhat unnatural, and it forces users to mark every single call -site instead of allowing them to mark everything within a statement at -once. - -Autoclosures pose a problem for marking. For the most part, we want -to pretend that the expression of an autoclosure is being evaluated in -the enclosing context; we don't want to have to mark both a call -within the autoclosure and the call to the function taking the -autoclosure! We should teach the type-checking pass to recognize this -pattern: a call to a function that ``throwsIf`` an autoclosure argument -does. - -There's a similar problem with functions that are supposed to feel -like statements. We want you to be able to write:: - - autoreleasepool { - let string = parseString(try) - ... - } - -without marking the call to ``autoreleasepool``, because this undermines -the ability to write functions that feel like statements. However, -there are other important differences between these trailing-closure -uses and true built-in statements, such as the behavior of ``return``, -``break``, and ``continue``. An attribute which marks the function as -being statement-like would be a necessary step towards addressing both -problems. Doing this reliably in closures would be challenging, -however. - -Asserting markers -~~~~~~~~~~~~~~~~~ - -Typed propagation is a hypothesis-checking mechanism and so suffers -from the standard problem of false positives. (Basic soundness -eliminates false negatives, of course: the compiler is supposed to -force programmers to deal with *every* source of error.) In this -case, a false positive means a situation where an API is declared to -throw but an error is actually dynamically impossible. - -For example, a function to load an image from a URL would usually be -designed to produce an error if the image didn't exist, the connection -failed, the file data was malformed, or any of a hundred other -problems arose. The programmer should be expected to deal with that -error in general. But a programmer might reasonably use the same API -to load an image completely under their control, e.g. from their -program's private resources. We shouldn't make it too syntactically -inconvenient to "turn off" error-checking for such calls. - -One important point is that we don't want to make it too easy to -*ignore* errors. Ignored errors usually lead to a terrible debugging -experience, even if the error is logged with a meaningful stack trace; -the full context of the failure is lost and can be difficult to -reproduce. Ignored errors also have a way of compounding, where an -error that's "harmlessly" ignored at one layer of abstraction causes -another error elsewhere; and of course the second error can be -ignored, etc., but only by making the program harder and harder to -understand and debug, leaving behind log files that are increasingly -jammed with the detritus of a hundred ignored errors. And finally, -ignoring errors creates a number of type-safety and security problems -by encouraging programs to blunder onwards with meaningless data and -broken invariants. - -Instead, we just want to make it (comparatively) easy to turn a static -problem into a dynamic one, much as assertions and the ! operator do. -Of course, this needs to be an explicit operation, because otherwise -we would completely lose typed propagation; and it should be -call-specific, so that the programmer has to make an informed decision -about individual operations. But we already have an explicit, -call-site-specific annotation: the ``try`` operator. So the obvious -solution is to allow a variant of ``try`` that asserts that an error -is not thrown out of its operand; and the obvious choice there within -our existing design language is to use the universal "be careful, this -is unsafe" marker by making the keyword ``try!``. - -It's reasonable to ask whether ``try!`` is actually *too* easy to -write, given that this is, after all, an unsafe operation. One quick -rejoinder is that it's no worse than the ordinary ``!`` operator in -that sense. Like ``!``, it's something that a cautious programmer -might want to investigate closer, and you can easily imagine codebases -that expect uses of it to always be explained in comments. But more -importantly, just like ``!`` it's only *statically* unsafe, and it -will reliably fail when the programmer is wrong. Therefore, while you -can easily imagine (and demonstrate) uncautious programmers flailing -around with it to appease the type-checker, that's not actually a -tenable position for the overall program: eventually the programmer -will have to learn how to use the feature, or else their program -simply won't run. - -Furthermore, while ``try!`` does somewhat undermine error-safety in -the hands of a careless programmer, it's still better to promote this -kind of unsafety than to implicitly promote the alternative. A -careless programmer isn't going to write good error handling just -because we don't give them this feature. Instead, they'll write out a -``do/catch`` block, and the natural pressure there will be to silently -swallow the error --- after all, that takes less boilerplate than -asserting or logging. - -In a future release, when we add support for universal errors, we'll -need to reconsider the behavior of ``try!``. One possibility is that -``try!`` should simply start propagating its operand as a universal -error; this would allow emergency recovery. Alternatively, we may -want ``try!`` to assert that even universal errors aren't thrown out -of it; this would provide a more consistent language model between the -two kinds of errors. But we don't need to think too hard about this -yet. - -Other syntax ------------- - -Clean-up actions -~~~~~~~~~~~~~~~~ - -Swift should provide a statement for cleaning up with an *ad hoc* -action. - -Overall, I think it is better to use a Go-style ``defer`` than a -Java-style ``try ... finally``. While this makes the exact order of -execution more obscure, it does make it obvious that the clean-up -*will* be executed without any further analysis, which is something -that readers will usually be interested in. - -Unlike Go, I think this should be tied to scope-exit, not to -function-exit. This makes it very easy to know the set of ``defer`` -actions that will be executed when a scope exits: it's all the ``defer`` -statement in exactly that scope. In contrast, in Go you have to -understand the dynamic history of the function's execution. This also -eliminates some semantic and performance oddities relating to variable -capture, since the ``defer`` action occurs with everything still in -scope. One downside is that it's not as good for "transactional" -idioms which push an undo action for everything they do, but that -style has composition problems across function boundaries anyway. - -I think ``defer`` is a reasonable name for this, although we might also -consider ``finally``. I'll use ``defer`` in the rest of this proposal. - -``defer`` may be followed by an arbitrary statement. The compiler -should reject an action that might terminate early, whether by -throwing or with ``return``, ``break``, or ``continue``. - -Examples:: - - if exists(filename) { - let file = open(filename, O_READ) - defer close(file) - - while let line = try file.readline() { - ... - } - - // close occurs here, at the end of the formal scope. - } - -We should consider providing a convenient way to mark that a ``defer`` -action should only be taken if an error is thrown. This is a -convenient shorthand for controlling the action with a flag that's -only set to true at the end of an operation. The flag approach is -often more useful, since it allows the action to be taken for *any* -early exit, e.g. a ``return``, not just for error propagation. - -``using`` -~~~~~~~~~ - -Swift should consider providing a ``using`` statement which acquires a -resource, holds it for a fixed period of time, optionally binds it to -a name, and then releases it whenever the controlled statement exits. - -``using`` has many similarities to ``defer``. It does not subsume -``defer``, which is useful for many ad-hoc and tokenless clean-ups. But -it is convenient for the common pattern of a type-directed clean-up. - -We do not expect this feature to be necessary in the first release. - -C and Objective-C Interoperation --------------------------------- - -It's of paramount importance that Swift's error model interact as -cleanly with Objective-C APIs as we can make it. - -In general, we want to try to import APIs that produce errors as -throwing; if this fails, we'll import the API as an ordinary -non-throwing function. This is a safe approach only under the -assumption that importing the function as throwing will require -significant changes to the call. That is, if a developer writes code -assuming that an API will be imported as throwing, but in fact Swift -fails to import the API that way, it's important that the code doesn't -compile. - -Fortunately, this is true for the common pattern of an error -out-parameter: if Swift cannot import the function as throwing, it -will leave the out-parameter in place, and the compiler will complain -if the developer fails to pass an error argument. However, it is -possible to imagine APIs where the "meat" of the error is returned in -a different way; consider a POSIX API that simply sets ``errno``. Great -care would need to be taken when such an API is only partially -imported as throwing. - -Let's wade into the details. - -Error types -~~~~~~~~~~~ - -``NSError`` and ``CFError`` should implement the ``ErrorType`` protocol. It -should be possible to turn an arbitrary Swift ``enum`` that conforms to -``ErrorType`` into an ``NSError`` by using the qualified type name as the -domain key, the enumerator as the error code, and turning the payload -into user data. - -Recognizing system enums as error domains is a matter of annotation. -Most likely, Swift will just special-case a few common domains in -the first release. - -Objective-C method error patterns -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The most common error pattern in ObjC by far is for a method to have -an autoreleased ``NSError**`` out-parameter. We don't currently propose -automatically importing anything as ``throws`` when it lacks such a -parameter. - -If any APIs take an ``NSError**`` and *don't* intend for it to be an -error out-parameter, they will almost certainly need it to be marked. - -Detecting an error -^^^^^^^^^^^^^^^^^^ - -Many of these methods have some sort of significant result which -is used for testing whether an error occurred: - -* The most common pattern is a ``BOOL`` result, where a false value - means an error occurred. This seems unambiguous. - - Swift should import these methods as if they'd returned ``Void``. - -* Also common is a pointer result, where a ``nil`` result usually means - an error occurred. - - I've been told that there are some exceptions to this rule, where a - ``nil`` result is valid and the caller is apparently meant to check - for a non-``nil`` error. I haven't been able to find any such APIs - in Cocoa, though; the claimed APIs I've been referred to do have - nullable results, but returned via out-parameters with a `BOOL` - formal result. So it seems to be a sound policy decision for - Objective-C that ``nil`` results are errors by default. CF might be - a different story, though. - - When a ``nil`` result implies that an error has occurred, Swift - should import the method as returning a non-optional result. - -* A few CF APIs return ``void``. As far as I can tell, for all of - these, the caller is expected to check for a non-``nil`` error. - -For other sentinel cases, we can consider adding a new clang attribute -to indicate to the compiler what the sentinel is: - -* There are several APIs returning ``NSInteger`` or ``NSUInteger``. At - least some of these return 0 on error, but that doesn't seem like a - reasonable general assumption. - -* ``AVFoundation`` provides a couple methods returning - ``AVKeyValueStatus``. These produce an error if the API returned - ``AVKeyValueStatusFailed``, which, interestingly enough, is not the - zero value. - -The clang attribute would specify how to test the return value for -an error. For example:: - - + (NSInteger)writePropertyList:(id)plist - toStream:(NSOutputStream *)stream - format:(NSPropertyListFormat)format - options:(NSPropertyListWriteOptions)opt - error:(out NSError **)error - NS_ERROR_RESULT(0) - - - (AVKeyValueStatus)statusOfValueForKey:(NSString *)key - error:(NSError **) - NS_ERROR_RESULT(AVKeyValueStatusFailed); - -We should also provide a Clang attribute which specifies that the -correct way to test for an error is to check the out-parameter. Both -of these attributes could potentially be used by the static analyzer, -not just Swift. (For example, they could try to detect an invalid -error check.) - -A constant value would be sufficient for the cases I've seen, but if -the argument has to generalized to a simple expression, that's still -feasible. - -The error parameter -^^^^^^^^^^^^^^^^^^^ - -The obvious import rule for Objective-C methods with ``NSError**`` -out-parameters is to simply mark them ``throws`` and remove the selector -clause corresponding to the out-parameter. That is, a method like -this one from ``NSAttributedString``:: - - - (NSData *)dataFromRange:(NSRange)range - documentAttributes:(NSDictionary *)dict - error:(NSError **)error; - -would be imported as:: - - func dataFromRange(range: NSRange, - documentAttributes dict: NSDictionary) throws -> NSData - -However, applying this rule haphazardly causes problems for -Objective-C interoperation, because multiple methods can be imported -the same way. The model is far more comprehensible to both compiler -and programmer if the original Objective-C declaration can be -unambiguously reconstructed from a Swift declaration. - -There are two sources of this ambiguity: - -* The error parameter could have appeared at an arbitrary position in - the selector; that is, both ``foo:bar:error:`` and ``foo:error:bar:`` - would appear as ``foo:bar:`` after import. - -* The error parameter could have had an arbitrary selector chunk; - that is, both ``foo:error:`` and ``foo:withError:`` would appear as - ``foo:`` after import. - -To allow reconstruction, then, we should only apply the rule when the -error parameter is the last parameter and the corresponding selector -is either ``error:`` or the first chunk. Empirically, this seems to do -the right thing for all but two sets of APIs in the public API: - -* The ``ISyncSessionDriverDelegate`` category on ``NSObject`` declares - half-a-dozen methods like this:: - - - (BOOL)sessionDriver:(ISyncSessionDriver *)sender - didRegisterClientAndReturnError:(NSError **)outError; - - Fortunately, these delegate methods were all deprecated in Lion, - and Swift currently doesn't even import deprecated methods. - -* ``NSFileCoordinator`` has half a dozen methods where the ``error:`` - clause is second-to-last, followed by a block argument. These - methods are not deprecated as far as I know. - -Of course, user code could also fail to follow this rule. - -I think it's acceptable for Swift to just not import these methods as -``throws``, leaving the original error parameter in place exactly as if -they didn't follow an intelligible pattern in the header. - -This translation rule would import methods like this one from -``NSDocument``:: - - - (NSDocument *)duplicateAndReturnError:(NSError **)outError; - -like so:: - - func duplicateAndReturnError() throws -> NSDocument - -Leaving the ``AndReturnError`` bit around feels unfortunate to me, but I -don't see what we could do without losing the ability to automatically -reconstruct the Objective-C signature. This pattern is common but -hardly universal; consider this method from ``NSManagedObject``:: - - - (BOOL)validateForDelete:(NSError **)error; - -This would be imported as:: - - func validateForDelete() throws - -This seems like a really nice import. - -CoreFoundation functions -~~~~~~~~~~~~~~~~~~~~~~~~ - -CF APIs use ``CFErrorRef`` pretty reliably, but there are two problems. - -First, we're not as confident about the memory management rules for -the error object. Is it always returned at +1? - -Second, I'm not as confident about how to detect that an error has -occurred: - -* There are a lot of functions that return ``Boolean`` or ``bool``. It's - likely that these functions consistently use the same convention as - Objective-C: false means error. - -* Similarly, there are many functions that return an object reference. - Again, we'd need a policy on whether to treat ``nil`` results as - errors. - -* There are a handful of APIs that return a ``CFIndex``, all with - apparently the same rule that a zero value means an error. (These - are serialization APIs, so writing nothing seems like a reasonable - error.) But just like Objective-C, that does not seem like a - reasonable default assumption. - -* ``ColorSyncProfile`` has several related functions that return - ``float``! These are both apparently meant to be checked by testing - whether the error result was filled in. - -There are also some APIs that do not use ``CFErrorRef``. For example, -most of the ``CVDisplayLink`` APIs in CoreVideo returns their own -``CVReturn`` enumeration, many with more than one error value. -Obviously, these will not be imported as throwing unless CoreVideo -writes an overlay. - -Other C APIs -~~~~~~~~~~~~ - -In principle, we could import POSIX functions into Swift as throwing -functions, filling in the error from ``errno``. It's nearly impossible -to imagine doing this with an automatic import rule, however; much -more likely, we'd need to wrap them all in an overlay. - -Implementation design ---------------------- - -Error propagation for the kinds of explicit, typed errors that I've -been focusing on should be handled by implicit manual propagation. It -would be good to bias the implementation somewhat towards the -non-error path, perhaps by moving error paths to the ends of functions -and so on, and perhaps even by processing cleanups with an -interpretive approach instead of directly inlining that code, but we -should not bias so heavily as to seriously compromise performance. In -other words, we should not use table-based unwinding. - -Error propagation for universal errors should be handled by -table-based unwinding. ``catch`` handlers can catch both, mapping -unwind exceptions to ``ErrorType`` values as necessary. With a -carefully-designed interpretation function aimed to solve the specific -needs of Swift, we can avoid most of the code-size impact by shifting -it to the unwind tables, which needn't ever be loaded in the common -case. diff --git a/docs/Failable Initializers.md b/docs/Failable Initializers.md new file mode 100644 index 0000000000000..fef854f7c4f77 --- /dev/null +++ b/docs/Failable Initializers.md @@ -0,0 +1,206 @@ +This proposal describes the work required to address +rdar://problem/18216578. + +Some terminology used below: + +- **deallocating** refers to freeing the memory of an object without + running any destructors. +- **releasing** refers to giving up a reference, which will result in + running the destructor and deallocation of the object if this was + the last reference. +- A **destructor** is a Swift-generated entry point which call the + user-defined deinitializer, then releases all stored properties. +- A **deinitializer** is an optional user-defined entry point in a + Swift class which handles any necessary cleanup beyond releasing + stored properties. +- A **slice** of an object is the set of stored properties defined in + one particular class forming the superclass chain of the instance. + +Failable initializers +===================== + +A **failable initializer** can return early with an error, without +having initialized a new object. Examples can include initializers which +validate input arguments, or attempt to acquire a limited resource. + +There are two types of failable initializers: + +- An initializer can be declared as having an optional return type, in + which case it can signal failure by returning nil. +- An initializer can be declared as throwing, in which case it can + signal failure by throwing an error. + +Convenience initializers +------------------------ + +Failing convenience initializers are the easy case, and are fully +supported now. The failure can occur either before or after the +self.init() delegation, and is handled as follows: + +> 1. A failure prior to the self.init() delegation is handled by +> deallocating the completely-uninitialized self value. +> 2. A failure after the self.init() delegation is handled by releasing +> the fully-initialized self.value. + +Designated initializers +----------------------- + +Failing designated initializers are more difficult, and are the subject +of this proposal. + +Similarly to convenience initializers, designated initializers can fail +either before or after the super.init() delegation (or, for a root class +initializer, the first location where all stored properties become +initialized). + +When failing after the super.init() delegation, we already have a +fully-initialized self value, so releasing the self value is sufficient. +The user-defined deinitializer, if any, is run in this case. + +A failure prior to the super.init() delegation on the other hand will +leave us with a partially-initialized self value that must be +deallocated. We have to deinitialize any stored properties of self that +we initialized, but we do not invoke the user-defined deinitializer +method. + +Description of the problem +-------------------------- + +To illustrate, say we are constructing an instance of a class C, and let +superclasses(C) be the sequence of superclasses, starting from C and +ending at a root class C\_n: + + superclasses(C) = {C, C_1, C_2, ..., C_n} + +Suppose our failure occurs in the designated initializer for class C\_k. +At this point, the self value looks like this: + +> 1. All stored properties in `{C, ..., C_(k-1)}` have +> been initialized. +> 2. Zero or more stored properties in `C_k` have been initialized. +> 3. The rest of the object `{C_(k+1), ..., C_n}` is +> completely uninitialized. + +In order to fail out of the constructor without leaking memory, we have +to destroy the initialized stored properties only without calling any +Swift deinit methods, then deallocate the object itself. + +There is a further complication once we take Objective-C +interoperability into account. Objective-C classes can override -alloc, +to get the object from a memory pool, for example. Also, they can +override -retain and -release to implement their own reference counting. +This means that if our class has @objc ancestry, we have to release it +with -release even if it is partially initialized -- since this will +result in Swift destructors being called, they have to know to skip the +uninitialized parts of the object. + +There is an issue we need to sort out, tracked by rdar://18720947. +Basically, if we haven't done the `super.init()`, is it safe to call +`-release`. The rest of this proposal assumes the answer is "yes". + +Possible solutions +------------------ + +One approach is to think of the super.init() delegation as having a +tri-state return value, instead of two-state: + +> 1. First failure case -- object is fully initialized +> 2. Second failure case -- object is partially initialized +> 3. Success + +This is problematic because now the ownership conventions in the +initializer signature do not really describe the initializer's effect on +reference counts; we now that this special return value for the second +failure case, where the self value looks like it should have been +consumed but it wasn't. + +It is also difficult to encode this tri-state return for throwing +initializers. One can imagine changing the try\_apply and throw SIL +instructions to support returning a pair (ErrorType, AnyObject) instead +of a single ErrorType. But this would ripple changes throughout various +SIL analyses, and require IRGen to encode the pair return value in an +efficient way. + +Proposed solution -- pure Swift case +------------------------------------ + +A simpler approach seems to be to introduce a new partialDeinit entry +point, referenced through a special kind of SILDeclRef. This entry point +is dispatched through the vtable and invoked using a standard +class\_method sequence in SIL. + +This entry point's job is to conditionally deinitialize stored +properties of the self value, without invoking the user-defined +deinitializer. + +When a designated initializer for class C\_k fails prior to performing +the super.init() delegation, we emit the following code sequence: + +> 1. First, de-initialize any stored properties this initializer may +> have initialized. +> 2. Second, invoke `partialDeinit(self, M)`, where M is the static +> metatype of `C_k`. + +The partialDeinit entry point is implemented as follows: + +> 1. If the static self type of the entry point is not equal to M, +> first delegate to the superclass's partialDeinit entry point, then +> deinitialize all stored properties in `C_k`. +> 2. If the static self type is equal to M, we have finished +> deinitializing the object, and we can now call a runtime function +> to deallocate it. + +Note that we delegate to the superclass partialDeinit entry point before +doing our own deinitialization, to ensure that stored properties are +deinitialized in the reverse order in which they were initialized. This +might not matter. + +Note that if even if a class does not have any failing initializers of +its own, it might delegate to a failing initializer in its superclass, +using `super.init!` or `try!`. It might be easiest to emit a +partialDeinit entry point for all classes, except those without any +stored properties. + +Proposed solution -- Objective-C case +------------------------------------- + +As noted above, if the class has `@objc` ancestry, the interoperability +story becomes more complicated. In order to undo any custom logic +implemented in an Objective-C override of `-alloc` or `-retain`, we have +to free the partially-initialized object using `-release`. + +To ensure we don't double-free any Swift stored properties, we will add +a new hidden stored property to each class that directly defines failing +initializers. The bit is set if this slice of the instance has been +initialized. + +Note that unlike partailDeinit, if a class does not have failing +initializers, it does not need this bit, even if its initializer +delegates to a failing initializer in a superclass. + +If the bit is clear, the destructor will skip the slice and not call the +user-defined `deinit` method, or delegate further up the chain. Note +that since newly-allocated Objective-C objects are zeroed out, the +initial state of this bit indicates the slice is not initialized. + +The constructor will set the bit before delegating to `super.init()`. + +If a destructor fails before delegating to `super.init()`, it will call +the partialDeinit entry point as before, but then, release the instance +instead of deallocating it. + +A possible optimization would be not generate the bit if all stored +properties are POD, or retainable pointers. In the latter case, all zero +bits is a valid representation (all the swift\_retain/release entry +points in the runtime check for null pointers, at least for now). +However, we do not have to do this optimization right away. + +Implementation +-------------- + +The bulk of this feature would be driven from DI. Right now, DI only +implements failing designated initializers in their full generality for +structs -- we already have logic for tracking which stored properties +have been initialized, but the rest of the support for the partialDeinit +entry point, as well as the Objective-C concerns needs to be fleshed +out. diff --git a/docs/Failable Initializers.rst b/docs/Failable Initializers.rst deleted file mode 100644 index b6fcfb59caae6..0000000000000 --- a/docs/Failable Initializers.rst +++ /dev/null @@ -1,204 +0,0 @@ -:orphan: - -This proposal describes the work required to address -rdar://problem/18216578. - -Some terminology used below: - -- **deallocating** refers to freeing the memory of an object without running - any destructors. - -- **releasing** refers to giving up a reference, which will result in running - the destructor and deallocation of the object if this was the last - reference. - -- A **destructor** is a Swift-generated entry point which call the user-defined - deinitializer, then releases all stored properties. - -- A **deinitializer** is an optional user-defined entry point in a Swift class - which handles any necessary cleanup beyond releasing stored properties. - -- A **slice** of an object is the set of stored properties defined in one - particular class forming the superclass chain of the instance. - -Failable initializers -===================== - -A **failable initializer** can return early with an error, without having -initialized a new object. Examples can include initializers which validate -input arguments, or attempt to acquire a limited resource. - -There are two types of failable initializers: - -- An initializer can be declared as having an optional return type, in - which case it can signal failure by returning nil. - -- An initializer can be declared as throwing, in which case it can signal - failure by throwing an error. - -Convenience initializers ------------------------- - -Failing convenience initializers are the easy case, and are fully supported -now. The failure can occur either before or after the self.init() -delegation, and is handled as follows: - - #. A failure prior to the self.init() delegation is handled by deallocating - the completely-uninitialized self value. - - #. A failure after the self.init() delegation is handled by releasing the - fully-initialized self.value. - -Designated initializers ------------------------ - -Failing designated initializers are more difficult, and are the subject of this -proposal. - -Similarly to convenience initializers, designated initializers can fail either -before or after the super.init() delegation (or, for a root class initializer, -the first location where all stored properties become initialized). - -When failing after the super.init() delegation, we already have a -fully-initialized self value, so releasing the self value is sufficient. The -user-defined deinitializer, if any, is run in this case. - -A failure prior to the super.init() delegation on the other hand will leave us -with a partially-initialized self value that must be deallocated. We have to -deinitialize any stored properties of self that we initialized, but we do -not invoke the user-defined deinitializer method. - -Description of the problem --------------------------- - -To illustrate, say we are constructing an instance of a class C, and let -superclasses(C) be the sequence of superclasses, starting from C and ending -at a root class C_n: - -:: - - superclasses(C) = {C, C_1, C_2, ..., C_n} - -Suppose our failure occurs in the designated initializer for class C_k. At this -point, the self value looks like this: - - #. All stored properties in ``{C, ..., C_(k-1)}`` have been initialized. - #. Zero or more stored properties in ``C_k`` have been initialized. - #. The rest of the object ``{C_(k+1), ..., C_n}`` is completely uninitialized. - -In order to fail out of the constructor without leaking memory, we have to -destroy the initialized stored properties only without calling any Swift -deinit methods, then deallocate the object itself. - -There is a further complication once we take Objective-C interoperability into -account. Objective-C classes can override -alloc, to get the object from a -memory pool, for example. Also, they can override -retain and -release to -implement their own reference counting. This means that if our class has @objc -ancestry, we have to release it with -release even if it is partially -initialized -- since this will result in Swift destructors being called, they -have to know to skip the uninitialized parts of the object. - -There is an issue we need to sort out, tracked by rdar://18720947. Basically, -if we haven't done the ``super.init()``, is it safe to call ``-release``. The -rest of this proposal assumes the answer is "yes". - -Possible solutions ------------------- - -One approach is to think of the super.init() delegation as having a tri-state -return value, instead of two-state: - - #. First failure case -- object is fully initialized - #. Second failure case -- object is partially initialized - #. Success - -This is problematic because now the ownership conventions in the initializer -signature do not really describe the initializer's effect on reference counts; -we now that this special return value for the second failure case, where the -self value looks like it should have been consumed but it wasn't. - -It is also difficult to encode this tri-state return for throwing initializers. -One can imagine changing the try_apply and throw SIL instructions to support -returning a pair (ErrorType, AnyObject) instead of a single ErrorType. But -this would ripple changes throughout various SIL analyses, and require IRGen -to encode the pair return value in an efficient way. - -Proposed solution -- pure Swift case ------------------------------------- - -A simpler approach seems to be to introduce a new partialDeinit entry point, -referenced through a special kind of SILDeclRef. This entry point is dispatched -through the vtable and invoked using a standard class_method sequence in SIL. - -This entry point's job is to conditionally deinitialize stored properties -of the self value, without invoking the user-defined deinitializer. - -When a designated initializer for class C_k fails prior to performing the -super.init() delegation, we emit the following code sequence: - - #. First, de-initialize any stored properties this initializer may have - initialized. - #. Second, invoke ``partialDeinit(self, M)``, where M is the static - metatype of ``C_k``. - -The partialDeinit entry point is implemented as follows: - - #. If the static self type of the entry point is not equal to M, first - delegate to the superclass's partialDeinit entry point, then - deinitialize all stored properties in ``C_k``. - - #. If the static self type is equal to M, we have finished deinitializing - the object, and we can now call a runtime function to deallocate it. - -Note that we delegate to the superclass partialDeinit entry point before -doing our own deinitialization, to ensure that stored properties are -deinitialized in the reverse order in which they were initialized. This -might not matter. - -Note that if even if a class does not have any failing initializers of its -own, it might delegate to a failing initializer in its superclass, using -``super.init!`` or ``try!``. It might be easiest to emit a partialDeinit -entry point for all classes, except those without any stored properties. - -Proposed solution -- Objective-C case -------------------------------------- - -As noted above, if the class has ``@objc`` ancestry, the interoperability -story becomes more complicated. In order to undo any custom logic implemented -in an Objective-C override of ``-alloc`` or ``-retain``, we have to free the -partially-initialized object using ``-release``. - -To ensure we don't double-free any Swift stored properties, we will add -a new hidden stored property to each class that directly defines failing -initializers. The bit is set if this slice of the instance has been -initialized. - -Note that unlike partailDeinit, if a class does not have failing initializers, -it does not need this bit, even if its initializer delegates to a failing -initializer in a superclass. - -If the bit is clear, the destructor will skip the slice and not call the -user-defined ``deinit`` method, or delegate further up the chain. Note that -since newly-allocated Objective-C objects are zeroed out, the initial state -of this bit indicates the slice is not initialized. - -The constructor will set the bit before delegating to ``super.init()``. - -If a destructor fails before delegating to ``super.init()``, it will call -the partialDeinit entry point as before, but then, release the instance -instead of deallocating it. - -A possible optimization would be not generate the bit if all stored -properties are POD, or retainable pointers. In the latter case, all zero bits -is a valid representation (all the swift_retain/release entry points in the -runtime check for null pointers, at least for now). However, we do not have -to do this optimization right away. - -Implementation --------------- - -The bulk of this feature would be driven from DI. Right now, DI only implements -failing designated initializers in their full generality for structs -- we -already have logic for tracking which stored properties have been initialized, -but the rest of the support for the partialDeinit entry point, as well as the -Objective-C concerns needs to be fleshed out. diff --git a/docs/Generics.md b/docs/Generics.md new file mode 100644 index 0000000000000..18605fe377183 --- /dev/null +++ b/docs/Generics.md @@ -0,0 +1,905 @@ +Generics in Swift +================= + +Motivation +---------- + +Most types and functions in code are expressed in terms of a single, +concrete set of sets. Generics generalize this notion by allowing one to +express types and functions in terms of an abstraction over a (typically +unbounded) set of types, allowing improved code reuse. A typical example +of a generic type is a linked list of values, which can be used with any +type of value. In C++, this might be expressed as: + + template + class List { + public: + struct Node { + T value; + Node *next; + }; + + Node *first; + }; + +where List<Int>, List<String>, and List<DataRecord> +are all distinct types that provide a linked list storing integers, +strings, and DataRecords, respectively. Given such a data structure, one +also needs to be able to implement generic functions that can operate on +a list of any kind of elements, such as a simple, linear search +algorithm: + + template + typename List::Node *find(const List&list, const T& value) { + for (typename List::Node *result = list.first; result; result = result->next) + if (result->value == value) + return result; + + return 0; + } + +Generics are important for the construction of useful libraries, because +they allow the library to adapt to application-specific data types +without losing type safety. This is especially important for +foundational libraries containing common data structures and algorithms, +since these libraries are used across nearly every interesting +application. + +The alternatives to generics tend to lead to poor solutions: + +- Object-oriented languages tend to use "top" types (id in + Objective-C, java.lang.Object in pre-generics Java, etc.) for their + containers and algorithms, which gives up static type safety. Pre- + generics Java forced the user to introduce run-time-checked type + casts when interacting with containers (which is overly verbose), + while Objective-C relies on id's unsound implicit conversion + behavior to eliminate the need for casts. +- Many languages bake common data structures (arrays, + dictionaries, tables) into the language itself. This is unfortunate + both because it significantly increases the size of the core + language and because users then tend to use this limited set of data + structures for *every* problem, even when another (not-baked-in) + data structure would be better. + +Swift is intended to be a small, expressive language with great support +for building libraries. We'll need generics to be able to build those +libraries well. + +Goals +----- + +- Generics should enable the development of rich generic libraries + that feel similar to first-class language features +- Generics should work on any type, whether it is a value type or some + kind of object type +- Generic code should be almost as easy to write as non-generic code +- Generic code should be compiled such that it can be executed with + any data type without requiring a separate "instantiation" step +- Generics should interoperate cleanly with run-time polymorphism +- Types should be able to retroactively modified to meet the + requirements of a generic algorithm or data structure + +As important as the goals of a feature are the explicit non-goals, which +we don't want or don't need to support: + +- Compile-time "metaprogramming" in any form +- Expression-template tricks a la Boost.Spirit, POOMA + +Polymorphism +------------ + +Polymorphism allows one to use different data types with a uniform +interface. Overloading already allows a form of polymorphism ( ad hoc +polymorphism) in Swift. For example, given: + + func +(x : Int, y : Int) -> Int { add... } + func +(x : String, y : String) -> String { concat... } + +we can write the expression "x + y", which will work for both integers +and strings. + +However, we want the ability to express an algorithm or data structure +independently of mentioning any data type. To do so, we need a way to +express the essential interface that algorithm or data structure +requires. For example, an accumulation algorithm would need to express +that for any type T, one can write the expression "x + y" (where x and y +are both of type T) and it will produce another T. + +Protocols +--------- + +Most languages that provide some form of polymorphism also have a way to +describe abstract interfaces that cover a range of types: Java and C\# +interfaces, C++ abstract base classes, Objective-C protocols, Scala +traits, Haskell type classes, C++ concepts (briefly), and many more. All +allow one to describe functions or methods that are part of the +interface, and provide some way to re-use or extend a previous interface +by adding to it. We'll start with that core feature, and build onto it +what we need. + +In Swift, I suggest that we use the term protocol for this feature, +because I expect the end result to be similar enough to Objective-C +protocols that our users will benefit, and (more importantly) different +enough from Java/C\# interfaces and C++ abstract base classes that those +terms will be harmful. The term trait comes with the wrong connotation +for C++ programmers, and none of our users know Scala. + +In its most basic form, a protocol is a collection of function +signatures: + + protocol Document { + func title() -> String + } + +Document describes types that have a title() operation that accepts no +arguments and returns a String. Note that there is implicitly a 'self' +type, which is the type that conforms to the protocol itself. This +follows how most object-oriented languages describe interfaces, but +deviates from Haskell type classes and C++ concepts, which require +explicit type parameters for all of the types. We'll revisit this +decision later. + +Protocol Inheritance +-------------------- + +Composition of protocols is important to help programmers organize and +understand a large number of protocols and the data types that conform +to those protocols. For example, we could extend our Document protocol +to cover documents that support versioning: + + protocol VersionedDocument : Document { + func version() -> Int + } + +Multiple inheritance is permitted, allowing us to form a directed +acyclic graph of protocols: + + protocol PersistentDocument : VersionedDocument, Serializable { + func saveToFile(filename : path) + } + +Any type that conforms to PersistentDocument also conforms to +VersionedDocument, Document, and Serializable, which gives us +substitutability. + +Self Types +---------- + +Protocols thus far do not give us an easy way to express simple binary +operations. For example, let's try to write a Comparable protocol that +could be used to search for a generic find() operation: + + protocol Comparable { + func isEqual(other : ???) -> bool + } + +Our options for filling in ??? are currently very poor. We could use the +syntax for saying "any type" or "any type that is comparable", as one +must do most OO languages, including Java, C\#, and Objective-C, but +that's not expressing what we want: that the type of both of the +arguments be the same. This is sometimes referred to as the binary +method problem ( +has a discussion of this problem, including the solution I'm proposing +below). + +Neither C++ concepts nor Haskell type classes have this particular +problem, because they don't have the notion of an implicit 'Self' type. +Rather, they explicitly parameterize everything. In C++ concepts: + + concept Comparable { + bool T::isEqual(T); + } + +Java and C\# programmers work around this issue by parameterizing the +interface, e.g. (in Java): + + abstract class Comparable> { + public bool isEqual(THIS other); + } + +and then a class X that wants to be Comparable will inherit from +Comparable<X>. This is ugly and has a number of pitfalls; see + . + +Scala and Strongtalk have the notion of the 'Self' type, which +effectively allows one to refer to the eventual type of 'self' (which we +call 'self'). 'Self' (which we call 'Self' in Swift) allows us to +express the Comparable protocol in a natural way: + + protocol Comparable { + func isEqual(other : Self) -> bool + } + +By expressing Comparable in this way, we know that if we have two +objects of type T where T conforms to Comparable, comparison between +those two objects with isEqual is well-typed. However, if we have +objects of different types T and U, we cannot compare those objects with +isEqual even if both T and U are Comparable. + +Self types are not without their costs, particularly in the case where +Self is used as a parameter type of a class method that will be +subclassed. Here, the parameter type ends up being (implicitly) +covariant, which tightens up type-checking but may also force us into +more dynamic type checks. We can explore this separately; within +protocols, type-checking for Self is more direct. + +Associated Types +---------------- + +In addition to Self, a protocol's operations often need to refer to +types that are related to the type of 'Self', such as a type of data +stored in a collection, or the node and edge types of a graph. For +example, this would allow us to cleanly describe a protocol for +collections: + + protocol Collection { + typealias Element + func forEach(callback : (value : Element) -> void) + func add(value : Element) + } + +It is important here that a generic function that refers to a given type +T, which is known to be a collection, can access the associated types +corresponding to T. For example, one could implement an "accumulate" +operation for an arbitrary Collection, but doing so requires us to +specify some constraints on the Value type of the collection. We'll +return to this later. + +Operators, Properties, and Subscripting +--------------------------------------- + +As previously noted, protocols can contain both function requirements +(which are in effect requirements for instance methods) and associated +type requirements. Protocols can also contain operators, properties, and +subscript operators: + + protocol RandomAccessContainer : Collection { + var length : Int + func ==(lhs : Self, rhs : Self) + subscript (i : Int) -> Element + } + +Operator requirements can be satisfied by operator definitions, property +requirements can be satisfied by either variables or properties, and +subscript requirements can be satisfied by subscript operators. + +Conforming to a Protocol +------------------------ + +Thus far, we have not actually shown how a type can meet the +requirements of a protocol. The most syntactically lightweight approach +is to allow implicit conformance. This is essentially duck typing, where +a type is assumed to conform to a protocol if it meets the syntactic +requirements of the protocol. For example, given: + + protocol Shape { + func draw() + } + +One could write a Circle struct such as: + + struct Circle { + var center : Point + var radius : Int + + func draw() { + // draw it + } + } + +Circle provides a draw() method with the same input and result types as +required by the Shape protocol. Therefore, Circle conforms to Shape. + +Implicit protocol conformance is convenient, because it requires no +additional typing. However, it can run into some trouble when an entity +that syntactically matches a protocol doesn't provide the required +semantics. For example, Cowboys also know how to "draw!": + + struct Cowboy { + var gun : SixShooter + + func draw() { + // draw! + } + } + +It is unlikely that Cowboy is meant to conform to Shape, but the method +name and signatures match, so implicit conformance deduces that Cowboy +conforms to Shape. Random collisions between types are fairly rare. +However, when one is using protocol inheritance with fine- grained +(semantic or mostly-semantic) differences between protocols in the +hierarchy, they become more common. See + for +examples of this problem as it surfaced with C++ concepts. It is not +clear at this time whether we want implicit conformance in Swift: +there's no existing code to worry about, and explicit conformance +(described below) provides some benefits. + +Explicit Protocol Conformance +----------------------------- + +Type authors often implement types that are intended to conform to a +particular protocol. For example, if we want a linked-list type to +conform to Collection, we can specify that it is by adding a protocol +conformance annotation to the type: + + struct EmployeeList : Collection { // EmployeeList is a collection + typealias Element = T + func forEach(callback : (value : Element) -> void) { /* Implement this */ } + func add(value : Element) { /* Implement this */ } + } + +This explicit protocol conformance declaration forces the compiler to +check that EmployeeList actually does meet the requirements of the +Collection protocol. If we were missing an operation (say, forEach) or +had the wrong signature, the definition of 'EmployeeList' would be +ill-formed. Therefore, explicit conformance provides both documentation +for the user of EmployeeList and checking for the author and future +maintainers of EmployeeList. + +Any nominal type (such as an enum, struct, or class) can be specified to +conform to one or more protocols in this manner. Additionally, a +typealias can be specified to conform to one or more protocols, e.g.,: + + typealias NSInteger : Numeric = Int + +While not technically necessary due to retroactive modeling (below), +this can be used to document and check that a particular type alias does +in fact meet some basic, important requirements. Moreover, it falls out +of the syntax that places requirements on associated types. + +Retroactive Modeling +-------------------- + +When using a set of libraries, it's fairly common that one library +defines a protocol (and useful generic entities requiring that protocol) +while another library provides a data type that provides similar +functionality to that protocol, but under a different name. Retroactive +modeling is the process by which the type is retrofitted (without +changing the type) to meet the requirements of the protocol. + +In Swift, we provide support for retroactive modeling by allowing +extensions, e.g.,: + + extension String : Collection { + typealias Element = char + func forEach(callback : (value : Element) -> void) { /* use existing String routines to enumerate characters */ } + func add(value : Element) { self += value /* append character */ } + } + +Once an extension is defined, the extension now conforms to the +Collection protocol, and can be used anywhere a Collection is expected. + +Default Implementations +----------------------- + +The functions declared within a protocol are requirements that any type +must meet if it wants to conform to the protocol. There is a natural +tension here, then, between larger protocols that make it easier to +write generic algorithms, and smaller protocols that make it easier to +write conforming types. For example, should a Numeric protocol implement +all operations, e.g.,: + + protocol Numeric { + func +(lhs : Self, rhs : Self) -> Self + func -(lhs : Self, rhs : Self) -> Self + func +(x : Self) -> Self + func -(x : Self) -> Self + } + +which would make it easy to write general numeric algorithms, but +requires the author of some BigInt class to implement a lot of +functionality, or should the numeric protocol implement just the core +operations: + + protocol Numeric { + func +(lhs : Self, rhs : Self) -> Self + func -(x : Self) -> Self + } + +to make it easier to adopt the protocol (but harder to write numeric +algorithms)? Both of the protocols express the same thing +(semantically), because one can use the core operations (binary +, unary +-) to implement the other algorithms. However, it's far easier to allow +the protocol itself to provide default implementations: + + protocol Numeric { + func +(lhs : Self, rhs : Self) -> Self + func -(lhs : Self, rhs : Self) -> Self { return lhs + -rhs } + func +(x : Self) -> Self { return x } + func -(x : Self) -> Self + } + +This makes it easier both to implement generic algorithms (which can use +the most natural syntax) and to make a new type conform to the protocol. +For example, if we were to define only the core algorithms in our BigNum +type: + + struct BigNum : Numeric { + func +(lhs : BigNum, rhs : BigNum) -> BigNum { ... } + func -(x : BigNum) -> BigNum { ... } + } + +the compiler will automatically synthesize the other operations needed +for the protocol. Moreover, these operations will be available to uses +of the BigNum class as if they had been written in the type itself (or +in an extension of the type, if that feature is used), which means that +protocol conformance actually makes it easier to define types that +conform to protocols, rather than just providing additional checking. + +Subtype Polymorphism +-------------------- + +Subtype polymorphism is based on the notion of substitutability. If a +type S is a subtype of a type T, then a value of type S can safely be +used where a value of type T is expected. Object-oriented languages +typically use subtype polymorphism, where the subtype relationship is +based on inheritance: if the class Dog inherits from the class Animal, +then Dog is a subtype of Animal. Subtype polymorphism is generally +dynamic, in the sense that the substitution occurs at run-time, even if +it is statically type-checked. + +In Swift, we consider protocols to be types. A value of protocol type +has an existential type, meaning that we don't know the concrete type +until run-time (and even then it varies), but we know that the type +conforms to the given protocol. Thus, a variable can be declared with +type "Serializable", e.g.,: + + var x : Serializable = // value of any Serializable type + x.serialize() // okay: serialize() is part of the Serializable protocol + +Naturally, such polymorphism is dynamic, and will require boxing of +value types to implement. We can now see how Self types interact with +subtype polymorphism. For example, say we have two values of type +Comparable, and we try to compare them: + + var x : Comparable = ... + var y : Comparable = ... + if x.isEqual(y) { // well-typed? + } + +Whether x.isEqual(y) is well-typed is not statically determinable, +because the dynamic type of x may different from the dynamic type of y, +even if they are both comparable (e.g., one is an Int and the other a +String). It can be implemented by the compiler as a dynamic type check, +with some general failure mode (aborting, throwing an exception, etc.) +if the dynamic type check fails. + +To express types that meet the requirements of several protocols, one +can just create a new protocol aggregating those protocols: + + protocol SerializableDocument : Document, Serializable { } + var doc : SerializableDocument + print(doc.title()) // okay: title() is part of the Document protocol, so we can call it + doc.serialize(stout) // okay: serialize() is part of the Serializable protocol + +However, this only makes sense when the resulting protocol is a useful +abstraction. A SerializableDocument may or may not be a useful +abstraction. When it is not useful, one can instead use protocol<> +types to compose different protocols, e.g.,: + + var doc : protocol + +Here, doc has an existential type that is known to conform to both the +Document and Serializable protocols. This gives rise to a natural "top" +type, such that every type in the language is a subtype of "top". Java +has java.lang.Object, C\# has object, Objective-C has "id" (although +"id" is weird, because it is also convertible to everything; it's best +not to use it as a model). In Swift, the "top" type is simply an empty +protocol composition: + + typealias Any = protocol<> + + var value : Any = 17 // an any can hold an integer + value = "hello" // or a String + value = (42, "hello", Red) // or anything else + +Bounded Parametric Polymorphism +------------------------------- + +Parametric polymorphism is based on the idea of providing type +parameters for a generic function or type. When using that function or +type, one substitutes concrete types for the type parameters. Strictly +speaking, parametric polymorphism allows *any* type to be substituted +for a type parameter, but it's useless in practice because that means +that generic functions or types cannot do anything to the type +parameters: they must instead rely on first-class functions passed into +the generic function or type to perform any meaningful work. + +Far more useful (and prevalent) is bounded parametric polymorphism, +which allows the generic function or type to specify constraints +(bounds) on the type parameters. By specifying these bounds, it becomes +far easier to write and use these generic functions and types. Haskell +type classes, Java and C\# generics, C++ concepts, and many other +language features support bounded parametric polymorphism. + +Protocols provide a natural way to express the constraints of a generic +function in Swift. For example, one could define a generic linked list +as: + + struct ListNode { + var Value : T + enum NextNode { case Node : ListNode, End } + var Next : NextNode + } + + struct List { + var First : ListNode::NextNode + } + +This list works on any type T. One could then add a generic function +that inserts at the beginning of the list: + + func insertAtBeginning(list : List, value : T) { + list.First = ListNode(value, list.First) + } + +Expressing Constraints +---------------------- + +Within the type parameter list of a generic type or function (e.g., the +<T> in ListNode<T>), the 'T' introduces a new type parameter +and the (optional) ": type" names a protocol (or protocol composition) +to which 'T' must conform. Within the body of the generic type or +function, any of the functions or types described by the constraints are +available. For example, let's implement a find() operation on lists: + + func find(list : List, value : T) -> Int { + var index = 0 + var current + for (current = list.First; current is Node; current = current.Next) { + if current.Value.isEqual(value) { // okay: T is Comparable + return index + } + index = index + 1 + } + return -1 + } + +In addition to providing constraints on the type parameters, we also +need to be able to constrain associated types. To do so, we introduce +the notion of a "where" clause, which follows the signature of the +generic type or function. For example, let's generalize our find +algorithm to work on any ordered collection: + + protocol OrderedCollection : Collection { + func size() -> Int + func getAt(index : Int) -> Element // Element is an associated type + } + + func find( + collection : C, value : C.Element) -> Int + { + for index in 0...collection.size() { + if (collection.getAt(index) == value) { // okay: we know that C.Element is Comparable + return index + } + } + return -1 + } + +The where clause is actually the more general way of expressing +constraints, and the constraints expressed in the angle brackets (e.g., +<C : OrderedCollection>) are just sugar for a where clause. For +example, the above find() signature is equivalent to: + + func find( + collection : C, value : C.Element)-> Int + +Note that find<C> is shorthand for (and equivalent to) find<C : +Any>, since every type conforms to the Any protocol composition. + +There are two other important kinds of constraints that need to be +expressible. Before we get to those, consider a simple "Enumerator" +protocol that lets us describe an iteration of values of some given +value type: + + protocol Enumerator { + typealias Element + func isEmpty() -> Bool + func next() -> Element + } + +Now, we want to express the notion of an enumerable collection, which +provides a iteration, which we do by adding requirements into the +protocol: + + protocol EnumerableCollection : Collection { + typealias EnumeratorType : Enumerator + where EnumeratorType.Element == Element + func getEnumeratorType() -> EnumeratorType + } + +Here, we are specifying constraints on an associated type +(EnumeratorType must conform to the Enumerator protocol), by adding a +conformance clause (: Enumerator) to the associated type definition. We +also use a separate where clause to require that the type of values +produced by querying the enumerator is the same as the type of values +stored in the container. This is important, for example, for use with +the Comparable protocol (and any protocol using Self types), because it +maintains type identity within the generic function or type. + +Constraint Inference +-------------------- + +Generic types often constrain their type parameters. For example, a +SortedDictionary, which provides dictionary functionality using some +kind of balanced binary tree (as in C++'s std::map), would require that +its key type be Comparable: + + class SortedDictionary { + // ... + } + +Naturally, one any generic operation on a SortedDictionary<K,V> +would also require that K be Comparable, e.g.,: + + func forEachKey(c : SortedDictionary, + f : (Key) -> Void) { /* ... */ } + +However, explicitly requiring that Key conform to Comparable is +redundant: one could not provide an argument for 'c' without the Key +type of the SortedDictionary conforming to Comparable, because the +SortedDictionary type itself could not be formed. Constraint inference +infers these additional constraints within a generic function from the +parameter and return types of the function, simplifying the +specification of forEachKey: + + func forEachKey(c : SortedDictionary, + f : (Key) -> Void) { /* ... */ } + +Type Parameter Deduction +------------------------ + +As noted above, type arguments will be deduced from the call arguments +to a generic function: + + var values : list + insertAtBeginning(values, 17) // deduces T = Int + +Since Swift already has top-down type inference (as well as the C++-like +bottom-up inference), we can also deduce type arguments from the result +type: + + func cast(value : T) -> U { ... } + var x : Any + var y : Int = cast(x) // deduces T = Any, U = Int + +We require that all type parameters for a generic function be deducible. +We introduce this restriction so that we can avoid introducing a syntax +for explicitly specifying type arguments to a generic function, e.g.,: + + var y : Int = cast(x) // not permitted: < is the less-than operator + +This syntax is horribly ambiguous in C++, and with good type argument +deduction, should not be necessary in Swift. + +Implementation Model +-------------------- + +Because generics are constrained, a well-typed generic function or type +can be translated into object code that uses dynamic dispatch to perform +each of its operations on type parameters. This is in stark contrast to +the instantiation model of C++ templates, where each new set of template +arguments requires the generic function or type to be compiled again. +This model is important for scalability of builds, so that the time to +perform type-checking and code generation scales with the amount of code +written rather than the amount of code instantiated. Moreover, it can +lead to smaller binaries and a more flexible language (generic functions +can be "virtual"). + +The translation model is fairly simple. Consider the generic find() we +implemented for lists, above: + + func find(list : List, value : T) -> Int { + var index = 0 + var current = list.First + while current is ListNode { // now I'm just making stuff up + if current.value.isEqual(value) { // okay: T is Comparable + return index + } + current = current.Next + index = index + 1 + } + return -1 + } + +to translate this into executable code, we form a vtable for each of the +constraints on the generic function. In this case, we'll have a vtable +for Comparable T. Every operation within the body of this generic +function type-checks to either an operation on some concrete type (e.g., +the operations on Int), to an operation within a protocol (which +requires indirection through the corresponding vtable), or to an +operation on a generic type definition, all of which can be emitted as +object code. + +Specialization +-------------- + +This implementation model lends itself to optimization when we know the +specific argument types that will be used when invoking the generic +function. In this case, some or all of the vtables provided for the +constraints will effectively be constants. By specializing the generic +function (at compile-time, link-time, or (if we have a JIT) run-time) +for these types, we can eliminate the cost of the virtual dispatch, +inline calls when appropriate, and eliminate the overhead of the generic +system. Such optimizations can be performed based on heuristics, user +direction, or profile-guided optimization. + +Existential Types and Generics +------------------------------ + +Both existential types and generics depend on dynamic dispatching based +on protocols. A value of an existential type (say, Comparable) is a pair +(value, vtable). 'value' stores the current value either directly (if it +fits in the 3 words allocated to the value) or as a pointer to the boxed +representation (if the actual representation is larger than 3 words). By +itself, this value cannot be interpreted, because it's type is not known +statically, and may change due to assignment. The vtable provides the +means to manipulate the value, because it provides a mapping between the +protocols to which the existential type conforms (which is known +statically) to the functions that implementation that functionality for +the type of the value. The value, therefore, can only be safely +manipulated through the functions in this vtable. + +A value of some generic type T uses a similar implementation model. +However, the (value, vtable) pair is split apart: values of type T +contain only the value part (the 3 words of data), while the vtable is +maintained as a separate value that can be shared among all T's within +that generic function. + +Overloading +----------- + +Generic functions can be overloaded based entirely on constraints. For +example, consider a binary search algorithm: + + func binarySearch< + C : EnumerableCollection where C.Element : Comparable + >(collection : C, value : C.Element) + -> C.EnumeratorType + { + // We can perform log(N) comparisons, but EnumerableCollection + // only supports linear walks, so this is linear time + } + + protocol RandomAccessEnumerator : Enumerator { + // splits a range in half, returning both halves + func split() -> (Enumerator, Enumerator) + } + + func binarySearch< + C : EnumerableCollection + where C.Element : Comparable, + C.EnumeratorType: RandomAccessEnumerator + >(collection : C, value : C.Element) + -> C.EnumeratorType + { + // We can perform log(N) comparisons and log(N) range splits, + // so this is logarithmic time + } + +If binarySearch is called with a sequence whose range type conforms to +RandomAccessEnumerator, both of the generic functions match. However, +the second function is more specialized, because its constraints are a +superset of the constraints of the first function. In such a case, +overloading should pick the more specialized function. + +There is a question as to when this overloading occurs. For example, +binarySearch might be called as a subroutine of another generic function +with minimal requirements: + + func doSomethingWithSearch< + C : EnumerableCollection where C.Element : Ordered + >( + collection : C, value : C.Element + ) -> C.EnumeratorType + { + binarySearch(collection, value) + } + +At the time when the generic definition of doSomethingWithSearch is +type-checked, only the first binarySearch() function applies, since we +don't know that C.EnumeratorType conforms to RandomAccessEnumerator. +However, when doSomethingWithSearch is actually invoked, +C.EnumeratorType might conform to the RandomAccessEnumerator, in which +case we'd be better off picking the second binarySearch. This amounts to +run-time overload resolution, which may be desirable, but also has +downsides, such as the potential for run-time failures due to +ambiguities and the cost of performing such an expensive operation at +these call sites. Of course, that cost could be mitigated in hot generic +functions via the specialization mentioned above. + +Our current proposal for this is to decide statically which function is +called (based on similar partial-ordering rules as used in C++), and +avoid run-time overload resolution. If this proves onerous, we can +revisit the decision later. + +Parsing Issues +-------------- + +The use of angle brackets to supply arguments to a generic type, while +familiar to C++/C\#/Java programmers, cause some parsing problems. The +problem stems from the fact that '<', '>', and '>>' (the +latter of which will show up in generic types such as +Array<Array<Int>>) match the 'operator' terminal in the +grammar, and we wish to continue using this as operators. + +When we're in the type grammar, this is a minor inconvenience for the +parser, because code like this: + + var x : Array + +will essentially parse the type as: + + identifier operator Int operator + +and verify that the operators are '<' and '>', respectively. Cases +involving <> are more interesting, because the type of: + + var y : Array> + +is effectively parsed as: + + identifier operator identifier operator identifier operator operator + +by splitting the '>>' operator token into two '>' operator +tokens. + +However, this is manageable, and is already implemented for protocol +composition (protocol<>). The larger problem occurs at expression +context, where the parser cannot disambiguate the tokens: + + Matrix(10, 10) + +i.e.,: + + identifier operator identifier operator unspaced_lparen integer- literal comma integer-literal rparen + +which can be interpreted as either: + + (greater_than + (less_than + (declref Matrix) + (declref Double) + (tuple + (integer_literal 10) + (integer_literal 10))) + +or: + + (constructor Matrix + (tuple + (integer_literal 10) + (integer_literal 10))) + +Both Java and C\# have this ambiguity. C\# resolves the ambiguity by +looking at the token after the closing '>' to decide which way to go; +Java seems to do the same. We have a few options: + +1. Follow C\# and Java and implement the infinite lookahead needed to + make this work. Note that we have true ambiguities, because one + could make either of the above parse trees well-formed. +2. Introduce some kind of special rule for '<' like we have for '(', + such as: an identifier followed by an unspaced '<' is a type, + while an identifier followed by spacing and then '<' is an + expression, or +3. Pick some syntax other than angle brackets, which is not ambiguous. + Note that neither '(' nor '\[' work, because they too have + expression forms. +4. Disambiguate between the two parses semantically. + +We're going to try a variant of \#1, using a variation of the +disambiguation rule used in C\#. Essentially, when we see: + + identifier < + +we look ahead, trying to parse a type parameter list, until parsing the +type parameter list fails or we find a closing '>'. We then look +ahead an additional token to see if the closing '>' is followed by a +'(', '.', or closing bracketing token (since types are most commonly +followed by a constructor call or static member access). If parsing the +type parameter list succeeds, and the closing angle bracket is followed +by a '(', '.', or closing bracket token, then the '<...>' sequence +is parsed as a generic parameter list; otherwise, the '<' is parsed +as an operator. diff --git a/docs/Generics.rst b/docs/Generics.rst deleted file mode 100644 index 49854392c0d4b..0000000000000 --- a/docs/Generics.rst +++ /dev/null @@ -1,888 +0,0 @@ -.. _Generics: - -Generics in Swift -================= - -Motivation ----------- - -Most types and functions in code are expressed in terms of a single, concrete -set of sets. Generics generalize this notion by allowing one to express types -and functions in terms of an abstraction over a (typically unbounded) set of -types, allowing improved code reuse. A typical example of a generic type is a -linked list of values, which can be used with any type of value. In C++, this -might be expressed as:: - - template - class List { - public: - struct Node { - T value; - Node *next; - }; - - Node *first; - }; - -where List, List, and List are all distinct types that -provide a linked list storing integers, strings, and DataRecords, -respectively. Given such a data structure, one also needs to be able to -implement generic functions that can operate on a list of any kind of elements, -such as a simple, linear search algorithm:: - - template - typename List::Node *find(const List&list, const T& value) { - for (typename List::Node *result = list.first; result; result = result->next) - if (result->value == value) - return result; - - return 0; - } - -.. @test('compile', howmany = 'all', cmake_args = ['COMPILER', '${CMAKE_CXX_COMPILER}']) - -Generics are important for the construction of useful libraries, because they -allow the library to adapt to application-specific data types without losing -type safety. This is especially important for foundational libraries containing -common data structures and algorithms, since these libraries are used across -nearly every interesting application. - -The alternatives to generics tend to lead to poor solutions: - -* Object-oriented languages tend to use "top" types (id in Objective-C, - java.lang.Object in pre-generics Java, etc.) for their containers and - algorithms, which gives up static type safety. Pre- generics Java forced the - user to introduce run-time-checked type casts when interacting with containers - (which is overly verbose), while Objective-C relies on id's unsound implicit - conversion behavior to eliminate the need for casts. -* Many languages bake common data structures (arrays, dictionaries, tables) into - the language itself. This is unfortunate both because it significantly - increases the size of the core language and because users then tend to use - this limited set of data structures for *every* problem, even when another - (not-baked-in) data structure would be better. - -Swift is intended to be a small, expressive language with great support for -building libraries. We'll need generics to be able to build those libraries -well. - -Goals ------ - -* Generics should enable the development of rich generic libraries that feel - similar to first-class language features -* Generics should work on any type, whether it is a value type or some kind of - object type -* Generic code should be almost as easy to write as non-generic code -* Generic code should be compiled such that it can be executed with any data - type without requiring a separate "instantiation" step -* Generics should interoperate cleanly with run-time polymorphism -* Types should be able to retroactively modified to meet the requirements of a - generic algorithm or data structure - -As important as the goals of a feature are the explicit non-goals, which we -don't want or don't need to support: - -* Compile-time "metaprogramming" in any form -* Expression-template tricks a la Boost.Spirit, POOMA - -Polymorphism ------------- - -Polymorphism allows one to use different data types with a uniform -interface. Overloading already allows a form of polymorphism ( ad hoc -polymorphism) in Swift. For example, given:: - - func +(x : Int, y : Int) -> Int { add... } - func +(x : String, y : String) -> String { concat... } - -.. @example.replace('add...','return 1') - example.replace('concat...','return ""') - test() - -we can write the expression "x + y", which will work for both integers and -strings. - -However, we want the ability to express an algorithm or data structure -independently of mentioning any data type. To do so, we need a way to express -the essential interface that algorithm or data structure requires. For example, -an accumulation algorithm would need to express that for any type T, one can -write the expression "x + y" (where x and y are both of type T) and it will -produce another T. - -Protocols ---------- - -Most languages that provide some form of polymorphism also have a way to -describe abstract interfaces that cover a range of types: Java and C# -interfaces, C++ abstract base classes, Objective-C protocols, Scala traits, -Haskell type classes, C++ concepts (briefly), and many more. All allow one to -describe functions or methods that are part of the interface, and provide some -way to re-use or extend a previous interface by adding to it. We'll start with -that core feature, and build onto it what we need. - -In Swift, I suggest that we use the term protocol for this feature, because I -expect the end result to be similar enough to Objective-C protocols that our -users will benefit, and (more importantly) different enough from Java/C# -interfaces and C++ abstract base classes that those terms will be harmful. The -term trait comes with the wrong connotation for C++ programmers, and none of our -users know Scala. - -In its most basic form, a protocol is a collection of function signatures:: - - protocol Document { - func title() -> String - } - -Document describes types that have a title() operation that accepts no arguments -and returns a String. Note that there is implicitly a 'self' type, -which is the type that conforms to the protocol itself. This follows how most -object-oriented languages describe interfaces, but deviates from Haskell type -classes and C++ concepts, which require explicit type parameters for all of the -types. We'll revisit this decision later. - -Protocol Inheritance --------------------- - -Composition of protocols is important to help programmers organize and -understand a large number of protocols and the data types that conform to those -protocols. For example, we could extend our Document protocol to cover documents -that support versioning:: - - protocol VersionedDocument : Document { - func version() -> Int - } - -Multiple inheritance is permitted, allowing us to form a directed acyclic graph -of protocols:: - - protocol PersistentDocument : VersionedDocument, Serializable { - func saveToFile(filename : path) - } - -.. @example.prepend('struct path {} ; protocol Serializable {}') - test(howmany='all') - -Any type that conforms to PersistentDocument also conforms to VersionedDocument, -Document, and Serializable, which gives us substitutability. - -Self Types ----------- - -Protocols thus far do not give us an easy way to express simple binary -operations. For example, let's try to write a Comparable protocol that could be -used to search for a generic find() operation:: - - protocol Comparable { - func isEqual(other : ???) -> bool - } - -Our options for filling in ??? are currently very poor. We could use the syntax -for saying "any type" or "any type that is comparable", as one must do most OO -languages, including Java, C#, and Objective-C, but that's not expressing what -we want: that the type of both of the arguments be the same. This is sometimes -referred to as the binary method problem -(http://www.cis.upenn.edu/~bcpierce/papers/binary.ps has a discussion of this -problem, including the solution I'm proposing below). - -Neither C++ concepts nor Haskell type classes have this particular problem, -because they don't have the notion of an implicit 'Self' type. Rather, -they explicitly parameterize everything. In C++ concepts:: - - concept Comparable { - bool T::isEqual(T); - } - -.. @ignore() -.. We don't have a compiler for ConceptC++ - -Java and C# programmers work around this issue by parameterizing the -interface, e.g. (in Java):: - - abstract class Comparable> { - public bool isEqual(THIS other); - } - -.. @ignore() -.. This test just doesn't compile at the moment, but that represents a - bug in swift - -and then a class X that wants to be Comparable will inherit from -Comparable. This is ugly and has a number of pitfalls; see -http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6479372 . - -Scala and Strongtalk have the notion of the 'Self' type, which effectively -allows one to refer to the eventual type of 'self' (which we call -'self'). 'Self' (which we call 'Self' in Swift) allows us to express the -Comparable protocol in a natural way:: - - protocol Comparable { - func isEqual(other : Self) -> bool - } - -By expressing Comparable in this way, we know that if we have two objects of -type T where T conforms to Comparable, comparison between those two objects with -isEqual is well-typed. However, if we have objects of different types T and U, -we cannot compare those objects with isEqual even if both T and U are -Comparable. - -Self types are not without their costs, particularly in the case where Self is -used as a parameter type of a class method that will be subclassed. Here, the -parameter type ends up being (implicitly) covariant, which tightens up -type-checking but may also force us into more dynamic type checks. We can -explore this separately; within protocols, type-checking for Self is more -direct. - -Associated Types ----------------- - -In addition to Self, a protocol's operations often need to refer to types that -are related to the type of 'Self', such as a type of data stored in a -collection, or the node and edge types of a graph. For example, this would allow -us to cleanly describe a protocol for collections:: - - protocol Collection { - typealias Element - func forEach(callback : (value : Element) -> void) - func add(value : Element) - } - -It is important here that a generic function that refers to a given type T, -which is known to be a collection, can access the associated types corresponding -to T. For example, one could implement an "accumulate" operation for an -arbitrary Collection, but doing so requires us to specify some constraints on -the Value type of the collection. We'll return to this later. - -Operators, Properties, and Subscripting ---------------------------------------- - -As previously noted, protocols can contain both function requirements (which are -in effect requirements for instance methods) and associated type -requirements. Protocols can also contain operators, properties, and subscript -operators:: - - protocol RandomAccessContainer : Collection { - var length : Int - func ==(lhs : Self, rhs : Self) - subscript (i : Int) -> Element - } - -Operator requirements can be satisfied by operator definitions, property -requirements can be satisfied by either variables or properties, and subscript -requirements can be satisfied by subscript operators. - -Conforming to a Protocol ------------------------- - -Thus far, we have not actually shown how a type can meet the requirements of a -protocol. The most syntactically lightweight approach is to allow implicit -conformance. This is essentially duck typing, where a type is assumed to conform -to a protocol if it meets the syntactic requirements of the protocol. For -example, given:: - - protocol Shape { - func draw() - } - -One could write a Circle struct such as:: - - struct Circle { - var center : Point - var radius : Int - - func draw() { - // draw it - } - } - -Circle provides a draw() method with the same input and result types as required -by the Shape protocol. Therefore, Circle conforms to Shape. - -Implicit protocol conformance is convenient, because it requires no additional -typing. However, it can run into some trouble when an entity that syntactically -matches a protocol doesn't provide the required semantics. For example, Cowboys -also know how to "draw!":: - - struct Cowboy { - var gun : SixShooter - - func draw() { - // draw! - } - } - -It is unlikely that Cowboy is meant to conform to Shape, but the method name and -signatures match, so implicit conformance deduces that Cowboy conforms to -Shape. Random collisions between types are fairly rare. However, when one is -using protocol inheritance with fine- grained (semantic or mostly-semantic) -differences between protocols in the hierarchy, they become more common. See -http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2005/n1798.html for examples -of this problem as it surfaced with C++ concepts. It is not clear at this time -whether we want implicit conformance in Swift: there's no existing code to worry -about, and explicit conformance (described below) provides some benefits. - -Explicit Protocol Conformance ------------------------------ - -Type authors often implement types that are intended to conform to a particular -protocol. For example, if we want a linked-list type to conform to Collection, -we can specify that it is by adding a protocol conformance annotation to the -type:: - - struct EmployeeList : Collection { // EmployeeList is a collection - typealias Element = T - func forEach(callback : (value : Element) -> void) { /* Implement this */ } - func add(value : Element) { /* Implement this */ } - } - -This explicit protocol conformance declaration forces the compiler to check that -EmployeeList actually does meet the requirements of the Collection protocol. If -we were missing an operation (say, forEach) or had the wrong signature, the -definition of 'EmployeeList' would be ill-formed. Therefore, explicit -conformance provides both documentation for the user of EmployeeList and -checking for the author and future maintainers of EmployeeList. - -Any nominal type (such as an enum, struct, or class) can be specified to conform -to one or more protocols in this manner. Additionally, a typealias can be -specified to conform to one or more protocols, e.g.,:: - - typealias NSInteger : Numeric = Int - -While not technically necessary due to retroactive modeling (below), this can be -used to document and check that a particular type alias does in fact meet some -basic, important requirements. Moreover, it falls out of the syntax that places -requirements on associated types. - -Retroactive Modeling --------------------- - -When using a set of libraries, it's fairly common that one library defines a -protocol (and useful generic entities requiring that protocol) while another -library provides a data type that provides similar functionality to that -protocol, but under a different name. Retroactive modeling is the process by -which the type is retrofitted (without changing the type) to meet the -requirements of the protocol. - -In Swift, we provide support for retroactive modeling by allowing -extensions, e.g.,:: - - extension String : Collection { - typealias Element = char - func forEach(callback : (value : Element) -> void) { /* use existing String routines to enumerate characters */ } - func add(value : Element) { self += value /* append character */ } - } - -Once an extension is defined, the extension now conforms to the Collection -protocol, and can be used anywhere a Collection is expected. - -Default Implementations ------------------------ - -The functions declared within a protocol are requirements that any type must -meet if it wants to conform to the protocol. There is a natural tension here, -then, between larger protocols that make it easier to write generic algorithms, -and smaller protocols that make it easier to write conforming types. For -example, should a Numeric protocol implement all operations, e.g.,:: - - protocol Numeric { - func +(lhs : Self, rhs : Self) -> Self - func -(lhs : Self, rhs : Self) -> Self - func +(x : Self) -> Self - func -(x : Self) -> Self - } - -which would make it easy to write general numeric algorithms, but requires the -author of some BigInt class to implement a lot of functionality, or should the -numeric protocol implement just the core operations:: - - protocol Numeric { - func +(lhs : Self, rhs : Self) -> Self - func -(x : Self) -> Self - } - -to make it easier to adopt the protocol (but harder to write numeric -algorithms)? Both of the protocols express the same thing (semantically), -because one can use the core operations (binary +, unary -) to implement the -other algorithms. However, it's far easier to allow the protocol itself to -provide default implementations:: - - protocol Numeric { - func +(lhs : Self, rhs : Self) -> Self - func -(lhs : Self, rhs : Self) -> Self { return lhs + -rhs } - func +(x : Self) -> Self { return x } - func -(x : Self) -> Self - } - -This makes it easier both to implement generic algorithms (which can use the -most natural syntax) and to make a new type conform to the protocol. For -example, if we were to define only the core algorithms in our BigNum type:: - - struct BigNum : Numeric { - func +(lhs : BigNum, rhs : BigNum) -> BigNum { ... } - func -(x : BigNum) -> BigNum { ... } - } - -the compiler will automatically synthesize the other operations needed for the -protocol. Moreover, these operations will be available to uses of the BigNum -class as if they had been written in the type itself (or in an extension of the -type, if that feature is used), which means that protocol conformance actually -makes it easier to define types that conform to protocols, rather than just -providing additional checking. - -Subtype Polymorphism --------------------- - -Subtype polymorphism is based on the notion of substitutability. If a type S is -a subtype of a type T, then a value of type S can safely be used where a value -of type T is expected. Object-oriented languages typically use subtype -polymorphism, where the subtype relationship is based on inheritance: if the -class Dog inherits from the class Animal, then Dog is a subtype of -Animal. Subtype polymorphism is generally dynamic, in the sense that the -substitution occurs at run-time, even if it is statically type-checked. - -In Swift, we consider protocols to be types. A value of protocol type has an -existential type, meaning that we don't know the concrete type until run-time -(and even then it varies), but we know that the type conforms to the given -protocol. Thus, a variable can be declared with type "Serializable", e.g.,:: - - var x : Serializable = // value of any Serializable type - x.serialize() // okay: serialize() is part of the Serializable protocol - -Naturally, such polymorphism is dynamic, and will require boxing of value types -to implement. We can now see how Self types interact with subtype -polymorphism. For example, say we have two values of type Comparable, and we try -to compare them:: - - var x : Comparable = ... - var y : Comparable = ... - if x.isEqual(y) { // well-typed? - } - -Whether x.isEqual(y) is well-typed is not statically determinable, because the -dynamic type of x may different from the dynamic type of y, even if they are -both comparable (e.g., one is an Int and the other a String). It can be -implemented by the compiler as a dynamic type check, with some general failure -mode (aborting, throwing an exception, etc.) if the dynamic type check fails. - -To express types that meet the requirements of several protocols, one can just -create a new protocol aggregating those protocols:: - - protocol SerializableDocument : Document, Serializable { } - var doc : SerializableDocument - print(doc.title()) // okay: title() is part of the Document protocol, so we can call it - doc.serialize(stout) // okay: serialize() is part of the Serializable protocol - -However, this only makes sense when the resulting protocol is a useful -abstraction. A SerializableDocument may or may not be a useful abstraction. When -it is not useful, one can instead use protocol<> types to compose different -protocols, e.g.,:: - - var doc : protocol - -Here, doc has an existential type that is known to conform to both the Document -and Serializable protocols. This gives rise to a natural "top" type, such that -every type in the language is a subtype of "top". Java has java.lang.Object, C# -has object, Objective-C has "id" (although "id" is weird, because it is also -convertible to everything; it's best not to use it as a model). In Swift, the -"top" type is simply an empty protocol composition:: - - typealias Any = protocol<> - - var value : Any = 17 // an any can hold an integer - value = "hello" // or a String - value = (42, "hello", Red) // or anything else - -Bounded Parametric Polymorphism -------------------------------- - -Parametric polymorphism is based on the idea of providing type parameters for a -generic function or type. When using that function or type, one substitutes -concrete types for the type parameters. Strictly speaking, parametric -polymorphism allows *any* type to be substituted for a type parameter, but it's -useless in practice because that means that generic functions or types cannot do -anything to the type parameters: they must instead rely on first-class functions -passed into the generic function or type to perform any meaningful work. - -Far more useful (and prevalent) is bounded parametric polymorphism, which allows -the generic function or type to specify constraints (bounds) on the type -parameters. By specifying these bounds, it becomes far easier to write and use -these generic functions and types. Haskell type classes, Java and C# generics, -C++ concepts, and many other language features support bounded parametric -polymorphism. - -Protocols provide a natural way to express the constraints of a generic function -in Swift. For example, one could define a generic linked list as:: - - struct ListNode { - var Value : T - enum NextNode { case Node : ListNode, End } - var Next : NextNode - } - - struct List { - var First : ListNode::NextNode - } - -This list works on any type T. One could then add a generic function that -inserts at the beginning of the list:: - - func insertAtBeginning(list : List, value : T) { - list.First = ListNode(value, list.First) - } - -Expressing Constraints ----------------------- - -Within the type parameter list of a generic type or function (e.g., the in -ListNode), the 'T' introduces a new type parameter and the (optional) ": -type" names a protocol (or protocol composition) to which 'T' must -conform. Within the body of the generic type or function, any of the functions -or types described by the constraints are available. For example, let's -implement a find() operation on lists:: - - func find(list : List, value : T) -> Int { - var index = 0 - var current - for (current = list.First; current is Node; current = current.Next) { - if current.Value.isEqual(value) { // okay: T is Comparable - return index - } - index = index + 1 - } - return -1 - } - -In addition to providing constraints on the type parameters, we also need to be -able to constrain associated types. To do so, we introduce the notion of a -"where" clause, which follows the signature of the generic type or -function. For example, let's generalize our find algorithm to work on any -ordered collection:: - - protocol OrderedCollection : Collection { - func size() -> Int - func getAt(index : Int) -> Element // Element is an associated type - } - - func find( - collection : C, value : C.Element) -> Int - { - for index in 0...collection.size() { - if (collection.getAt(index) == value) { // okay: we know that C.Element is Comparable - return index - } - } - return -1 - } - -The where clause is actually the more general way of expressing constraints, -and the constraints expressed in the angle brackets (e.g., ) are just sugar for a where clause. For example, the -above find() signature is equivalent to:: - - func find( - collection : C, value : C.Element)-> Int - -Note that find is shorthand for (and equivalent to) find, since -every type conforms to the Any protocol composition. - -There are two other important kinds of constraints that need to be -expressible. Before we get to those, consider a simple "Enumerator" protocol that -lets us describe an iteration of values of some given value type:: - - protocol Enumerator { - typealias Element - func isEmpty() -> Bool - func next() -> Element - } - -Now, we want to express the notion of an enumerable collection, which provides a -iteration, which we do by adding requirements into the protocol:: - - protocol EnumerableCollection : Collection { - typealias EnumeratorType : Enumerator - where EnumeratorType.Element == Element - func getEnumeratorType() -> EnumeratorType - } - -Here, we are specifying constraints on an associated type (EnumeratorType must -conform to the Enumerator protocol), by adding a conformance clause (: Enumerator) -to the associated type definition. We also use a separate where clause to -require that the type of values produced by querying the enumerator is the same as -the type of values stored in the container. This is important, for example, for -use with the Comparable protocol (and any protocol using Self types), because it -maintains type identity within the generic function or type. - -Constraint Inference --------------------- - -Generic types often constrain their type parameters. For example, a -SortedDictionary, which provides dictionary functionality using some kind of -balanced binary tree (as in C++'s std::map), would require that its key type be -Comparable:: - - class SortedDictionary { - // ... - } - -Naturally, one any generic operation on a SortedDictionary would also require -that K be Comparable, e.g.,:: - - func forEachKey(c : SortedDictionary, - f : (Key) -> Void) { /* ... */ } - -However, explicitly requiring that Key conform to Comparable is redundant: one -could not provide an argument for 'c' without the Key type of the -SortedDictionary conforming to Comparable, because the SortedDictionary type -itself could not be formed. Constraint inference infers these additional -constraints within a generic function from the parameter and return types of the -function, simplifying the specification of forEachKey:: - - func forEachKey(c : SortedDictionary, - f : (Key) -> Void) { /* ... */ } - -Type Parameter Deduction ------------------------- - -As noted above, type arguments will be deduced from the call arguments to a -generic function:: - - var values : list - insertAtBeginning(values, 17) // deduces T = Int - -Since Swift already has top-down type inference (as well as the C++-like -bottom-up inference), we can also deduce type arguments from the result type:: - - func cast(value : T) -> U { ... } - var x : Any - var y : Int = cast(x) // deduces T = Any, U = Int - -We require that all type parameters for a generic function be deducible. We -introduce this restriction so that we can avoid introducing a syntax for -explicitly specifying type arguments to a generic function, e.g.,:: - - var y : Int = cast(x) // not permitted: < is the less-than operator - -This syntax is horribly ambiguous in C++, and with good type argument deduction, -should not be necessary in Swift. - -Implementation Model --------------------- - -Because generics are constrained, a well-typed generic function or type can be -translated into object code that uses dynamic dispatch to perform each of its -operations on type parameters. This is in stark contrast to the instantiation -model of C++ templates, where each new set of template arguments requires the -generic function or type to be compiled again. This model is important for -scalability of builds, so that the time to perform type-checking and code -generation scales with the amount of code written rather than the amount of code -instantiated. Moreover, it can lead to smaller binaries and a more flexible -language (generic functions can be "virtual"). - -The translation model is fairly simple. Consider the generic find() we -implemented for lists, above:: - - func find(list : List, value : T) -> Int { - var index = 0 - var current = list.First - while current is ListNode { // now I'm just making stuff up - if current.value.isEqual(value) { // okay: T is Comparable - return index - } - current = current.Next - index = index + 1 - } - return -1 - } - -to translate this into executable code, we form a vtable for each of the -constraints on the generic function. In this case, we'll have a vtable for -Comparable T. Every operation within the body of this generic function -type-checks to either an operation on some concrete type (e.g., the operations -on Int), to an operation within a protocol (which requires indirection through -the corresponding vtable), or to an operation on a generic type definition, all -of which can be emitted as object code. - -Specialization --------------- - -This implementation model lends itself to optimization when we know the specific -argument types that will be used when invoking the generic function. In this -case, some or all of the vtables provided for the constraints will effectively -be constants. By specializing the generic function (at compile-time, link-time, -or (if we have a JIT) run-time) for these types, we can eliminate the cost of -the virtual dispatch, inline calls when appropriate, and eliminate the overhead -of the generic system. Such optimizations can be performed based on heuristics, -user direction, or profile-guided optimization. - -Existential Types and Generics ------------------------------- - -Both existential types and generics depend on dynamic dispatching based on -protocols. A value of an existential type (say, Comparable) is a pair (value, -vtable). 'value' stores the current value either directly (if it fits in the 3 -words allocated to the value) or as a pointer to the boxed representation (if -the actual representation is larger than 3 words). By itself, this value cannot -be interpreted, because it's type is not known statically, and may change due to -assignment. The vtable provides the means to manipulate the value, because it -provides a mapping between the protocols to which the existential type conforms -(which is known statically) to the functions that implementation that -functionality for the type of the value. The value, therefore, can only be -safely manipulated through the functions in this vtable. - -A value of some generic type T uses a similar implementation model. However, -the (value, vtable) pair is split apart: values of type T contain only the value -part (the 3 words of data), while the vtable is maintained as a separate value -that can be shared among all T's within that generic function. - -Overloading ------------ - -Generic functions can be overloaded based entirely on constraints. For example, -consider a binary search algorithm:: - - func binarySearch< - C : EnumerableCollection where C.Element : Comparable - >(collection : C, value : C.Element) - -> C.EnumeratorType - { - // We can perform log(N) comparisons, but EnumerableCollection - // only supports linear walks, so this is linear time - } - - protocol RandomAccessEnumerator : Enumerator { - // splits a range in half, returning both halves - func split() -> (Enumerator, Enumerator) - } - - func binarySearch< - C : EnumerableCollection - where C.Element : Comparable, - C.EnumeratorType: RandomAccessEnumerator - >(collection : C, value : C.Element) - -> C.EnumeratorType - { - // We can perform log(N) comparisons and log(N) range splits, - // so this is logarithmic time - } - -If binarySearch is called with a sequence whose range type conforms to -RandomAccessEnumerator, both of the generic functions match. However, the second -function is more specialized, because its constraints are a superset of the -constraints of the first function. In such a case, overloading should pick the -more specialized function. - -There is a question as to when this overloading occurs. For example, -binarySearch might be called as a subroutine of another generic function with -minimal requirements:: - - func doSomethingWithSearch< - C : EnumerableCollection where C.Element : Ordered - >( - collection : C, value : C.Element - ) -> C.EnumeratorType - { - binarySearch(collection, value) - } - -At the time when the generic definition of doSomethingWithSearch is -type-checked, only the first binarySearch() function applies, since we don't -know that C.EnumeratorType conforms to RandomAccessEnumerator. However, when -doSomethingWithSearch is actually invoked, C.EnumeratorType might conform to the -RandomAccessEnumerator, in which case we'd be better off picking the second -binarySearch. This amounts to run-time overload resolution, which may be -desirable, but also has downsides, such as the potential for run-time failures -due to ambiguities and the cost of performing such an expensive operation at -these call sites. Of course, that cost could be mitigated in hot generic -functions via the specialization mentioned above. - -Our current proposal for this is to decide statically which function is called -(based on similar partial-ordering rules as used in C++), and avoid run-time -overload resolution. If this proves onerous, we can revisit the decision later. - -Parsing Issues --------------- - -The use of angle brackets to supply arguments to a generic type, while familiar -to C++/C#/Java programmers, cause some parsing problems. The problem stems from -the fact that '<', '>', and '>>' (the latter of which will show up in generic -types such as Array>) match the 'operator' terminal in the grammar, -and we wish to continue using this as operators. - -When we're in the type grammar, this is a minor inconvenience for the parser, -because code like this:: - - var x : Array - -will essentially parse the type as:: - - identifier operator Int operator - -and verify that the operators are '<' and '>', respectively. Cases -involving <> are more interesting, because the type of:: - - var y : Array> - -is effectively parsed as:: - - identifier operator identifier operator identifier operator operator - -by splitting the '>>' operator token into two '>' operator tokens. - -However, this is manageable, and is already implemented for protocol composition -(protocol<>). The larger problem occurs at expression context, where the parser -cannot disambiguate the tokens:: - - Matrix(10, 10) - -i.e.,:: - - identifier operator identifier operator unspaced_lparen integer- literal comma integer-literal rparen - -which can be interpreted as either:: - - (greater_than - (less_than - (declref Matrix) - (declref Double) - (tuple - (integer_literal 10) - (integer_literal 10))) - -or:: - - (constructor Matrix - (tuple - (integer_literal 10) - (integer_literal 10))) - -Both Java and C# have this ambiguity. C# resolves the ambiguity by looking at -the token after the closing '>' to decide which way to go; Java seems to do the -same. We have a few options: - -1. Follow C# and Java and implement the infinite lookahead needed to make this - work. Note that we have true ambiguities, because one could make either of - the above parse trees well-formed. - -2. Introduce some kind of special rule for '<' like we have for '(', such as: an - identifier followed by an unspaced '<' is a type, while an identifier - followed by spacing and then '<' is an expression, or - -3. Pick some syntax other than angle brackets, which is not ambiguous. Note - that neither '(' nor '[' work, because they too have expression forms. - -4. Disambiguate between the two parses semantically. - -We're going to try a variant of #1, using a variation of the disambiguation -rule used in C#. Essentially, when we see:: - - identifier < - -we look ahead, trying to parse a type parameter list, until parsing the type -parameter list fails or we find a closing '>'. We then look ahead an additional -token to see if the closing '>' is followed by a '(', '.', or closing bracketing -token (since types are most commonly followed by a constructor call or static -member access). If parsing the type parameter list succeeds, and the closing -angle bracket is followed by a '(', '.', or closing bracket token, then the -'<...>' sequence is parsed as a generic parameter list; otherwise, the '<' -is parsed as an operator. - -.. @ignore('all') diff --git a/docs/GitWorkflows.md b/docs/GitWorkflows.md new file mode 100644 index 0000000000000..42e854bf603c3 --- /dev/null +++ b/docs/GitWorkflows.md @@ -0,0 +1,292 @@ +Git Workflows +============= + +Purpose +------- + +Swift development has been based on SVN since its inception. As part of +the transition to Git this document helps to address questions about how +common SVN workflows we use today translate to their Git counterparts as +well as to discuss Git workflow practices we plan on having — at least +initially — after the Git transition. Notably we will follow a model +where commits to trunk — which is the ‘master’ branch in Git — has +commits land (in the common case) via rebasing instead of merging. This +model is open to evolution later, but this mimics the workflow we have +today with SVN. + +SVN -> GIT Workflows +======================= + +The general SVN workflow consists of the following commands: + +1. Checkout: This means checking out/setting up a new repository. +2. Update: Pulling the latest remote changes into a local repository. +3. Committing: Committing a change to the remote repository. +4. Reverting: Reverting a change from a remote repository. +5. Browsing: Looking at commits. + +This document will show how to translate these commands to Git and +additionally how to configure Git. It assumes that one is attempting to +manipulate a Git repository via bash in a terminal. A lot of information +since this is supposed to be a short, actionable guide. For more +information, please see the Git crash course guide for SVN users at +<> + +*NOTE* Whenever we say the Swift repository, we mean any repository in +the Swift project. + +Quicksetup (TLDR) +----------------- + +For those who do not want to read the full document, use the following +commands to perform a simple repo setup for the Swift repository: + + $ git config --global user.name "" + $ git config --global user.email "" + $ mkdir swift-source && cd swift-source + $ git clone + $ git clone + $ git clone + $ (cd swift && git config branch.autosetuprebase always) + $ git clone + $ git clone + +Then to commit a new commit to the remote swift repository: + + $ git commit + $ git push origin master + +and to pull new commits from the remote swift repository: + + $ git pull origin master + +In order to ease updating all repositories, consider using the script in +'./utils/update-checkout'. This will automate updating the repositories +in the proper way. + +Preliminary +----------- + +Before beginning, we need to perform some global configuration of Git. +Git includes a username/email of the committer in every commit. By +default this is the current logged in user and the hostname of the +machine. This is /not/ what one wants. We configure Git globally (i.e. +across all repositories) to have our proper name and email by running +the following commands: + + $ git config --global user.name "" + $ git config --global user.email "" + +Checkout +-------- + +Normally if one wishes to checkout a repository in SVN, one would use a +command like this: + + $ SVN co + +This would then checkout the latest revision from the repository +specified by 'repository url' and place it into the directory 'local +directory'. In Git, instead of checking out the repository, one clones +the repository. This is done as follows: + + $ git clone + +This will cause Git to clone the repository at 'repository url' and +check out the 'master' branch. The 'master' branch corresponds to +'trunk' in SVN. For more information about branching in Git please see +<> + +Before beginning to commit though, we /must/ perform some default +configuration of our repository to match the Swift repository default +configuration by enabling default rebasing. + +Repository Configuration (Enabling Default Rebasing) +---------------------------------------------------- + +Once we have cloned the repository, we need to configure the repository +for our use. Specifically we want to configure the swift repository so +that we rebase whenever we update the repository (see the update section +below for more details): + + $ git config branch.autosetuprebase always + +By default when updating, Git will attempt to merge the remote changes +and your local changes. Ignoring what that sentence means, this is not +an SVN-esque model. Instead we want any local changes that we have to be +applied on top of any new remote changes. The 'branch.autosetuprebase' +flag causes this to be done automatically when ever one updates the +local repository. + +Update +------ + +In SVN, one updates your local repository by running the update command: + + $ SVN update + +In Git, instead of performing SVN update, one pulls from the remote +repository: + + $ git pull --rebase origin master + +This will pull any new remote commits into your local repository and +then replay your current local commits on top of those new commits. + +By default the '--rebase' flag is not necessary for the Swift repository +because it is configured to always rebase by setting the +'branch.autosetuprebase' flag (see the section 'Repository Configuration +(Enabling Default Rebasing)' above). + +Commit +------ + +In SVN, committing always means submitting changes to a remote +repository. In Git, committing refers to the process of first telling +Git to track a change by staging the change and then committing all +staged changes into a change in the local repository. One can have many +such commits. Then when one is ready, one pushes the new local changes +to the remote repository. We go through these steps in more detail +below: + +In terms of replicating the SVN model, there are now two steps. In order +to commit changes one first stages a changed file using 'git add': + + $ git add + +Then once all changes that you want to be apart of the commit have been +staged, a commit is created in the local repository via the 'commit' +command: + + $ git commit + +As a shortcut to commit /all/ changes to local files that are already +being tracked by Git to the local repository, you can use the '-a' +command: + + $ git commit -a + +In both of these cases, an editor will pop up to accept a commit +message. To specify a short commit message at the commandline, you can +use the '-m' flag: + + $ git commit -m 'My great commit message.' + +In order to see the diff of changes that have not been staged, run the +command: + + $ git diff + +To see all changes that have been staged, run the command: + + $ git diff --staged + +To get a diff for a specific revision/path, perform the following +command: + + $ git diff + +In order to get a more concise view of the files that have staged and or +unstaged changes, run the command: + + $ git status + +In order to restore a file from the last revision, one uses the checkout +command: + + $ git checkout + +To restore a file to a specific revision, one must use a longer form of +the command: + + $ git checkout -- + +To unstage a file, one uses the 'reset' command: + + $ git reset + +This tells Git to reset '<path>' in the staging area to the top of +tree commit (which in Git is called 'HEAD'). In order to correct a +mistake, you can pass the 'amend' flag to Git: + + $ git commit --amend + +This will cause all staged changes to be merged into 'HEAD'. Once one +has made all the relevant commits, in order to push the changes to the +remote repository the 'push' command is used: + + $ git push origin master + +If a different committer has committed changes such that there are +remote commits that are not present locally, this will fail. In order to +get around this issue, perform: + + $ git pull --rebase origin master + +in order to pull the new remote commits and replay your new commits on +top. Then try to push again. See the 'Checkout' section above how to +configure the local swift repository to always rebase allowing you to +drop the '--rebase' flag. + +Revert +------ + +In SVN reverting a commit implies performing a reverse merge. In Git, +this is no longer true. Instead one now just uses the 'revert' command: + + $ git revert + +This will cause Git to perform the reverse merge of that revision for +you against HEAD and bring up a message window for you to write a commit +message. This will be autofilled in with the title of the commit that is +going to be reverted and the revision number of that commit like so: + + Revert "" + + This reverts commit . + +One can edit this message as one sees fit. Once this has been done, the +revert will become a normal commit in your repository like any other +commit. Thus to revert the commit in the remote repository, you need to +perform a Git push: + + $ git push origin master + +Browsing +-------- + +This section explains how one can view Git changes. In order to view a +history of all changes on a branch to the beginning of time use the +'log' command: + + $ git log + +This will for each commit show the following information: + + commit + Author: + Date: + + + +To see history starting at a specific commit use the following form of a +Git log command: + + $ git log + +To see a oneline summary that includes just the title of the commit and +its hash, pass the '--oneline' command: + + $ git log --oneline + +It will not show you what was actually changed in each commit. In order +to see what was actually changed in a commit, use the command 'show': + + $ git show + +This will show the aforementioned information shown by Git log, but +additionally will perform a diff against top of tree showing you the +contents of the change. To see the changes for a specific commit, pass +the revision to Git show: + + $ git show diff --git a/docs/GitWorkflows.rst b/docs/GitWorkflows.rst deleted file mode 100644 index 24e5ce8e23760..0000000000000 --- a/docs/GitWorkflows.rst +++ /dev/null @@ -1,279 +0,0 @@ -:orphan: - -Git Workflows -============= - -Purpose -------- - -Swift development has been based on SVN since its inception. As part of the -transition to Git this document helps to address questions about how common SVN -workflows we use today translate to their Git counterparts as well as to discuss -Git workflow practices we plan on having — at least initially — after the Git -transition. Notably we will follow a model where commits to trunk — which is -the ‘master’ branch in Git — has commits land (in the common case) via rebasing -instead of merging. This model is open to evolution later, but this mimics the -workflow we have today with SVN. - -SVN -> GIT Workflows -==================== - -The general SVN workflow consists of the following commands: - -1. Checkout: This means checking out/setting up a new repository. -2. Update: Pulling the latest remote changes into a local repository. -3. Committing: Committing a change to the remote repository. -4. Reverting: Reverting a change from a remote repository. -5. Browsing: Looking at commits. - -This document will show how to translate these commands to Git and additionally -how to configure Git. It assumes that one is attempting to manipulate a Git -repository via bash in a terminal. A lot of information since this is supposed -to be a short, actionable guide. For more information, please see the Git crash -course guide for SVN users at - -*NOTE* Whenever we say the Swift repository, we mean any repository in the -Swift project. - -Quicksetup (TLDR) ------------------ - -For those who do not want to read the full document, use the following commands -to perform a simple repo setup for the Swift repository:: - - $ git config --global user.name "" - $ git config --global user.email "" - $ mkdir swift-source && cd swift-source - $ git clone - $ git clone - $ git clone - $ (cd swift && git config branch.autosetuprebase always) - $ git clone - $ git clone - -Then to commit a new commit to the remote swift repository:: - - $ git commit - $ git push origin master - -and to pull new commits from the remote swift repository:: - - $ git pull origin master - -In order to ease updating all repositories, consider using the script in -'./utils/update-checkout'. This will automate updating the repositories in the -proper way. - -Preliminary ------------ - -Before beginning, we need to perform some global configuration of Git. Git -includes a username/email of the committer in every commit. By default this is -the current logged in user and the hostname of the machine. This is /not/ what -one wants. We configure Git globally (i.e. across all repositories) to have our -proper name and email by running the following commands:: - - $ git config --global user.name "" - $ git config --global user.email "" - -Checkout --------- - -Normally if one wishes to checkout a repository in SVN, one would use a command -like this:: - - $ SVN co - -This would then checkout the latest revision from the repository specified by -'repository url' and place it into the directory 'local directory'. In Git, -instead of checking out the repository, one clones the repository. This is done -as follows:: - - $ git clone - -This will cause Git to clone the repository at 'repository url' and check out -the 'master' branch. The 'master' branch corresponds to 'trunk' in SVN. For more -information about branching in Git please see - - -Before beginning to commit though, we /must/ perform some default configuration -of our repository to match the Swift repository default configuration by -enabling default rebasing. - -Repository Configuration (Enabling Default Rebasing) ----------------------------------------------------- - -Once we have cloned the repository, we need to configure the repository for our -use. Specifically we want to configure the swift repository so that we rebase -whenever we update the repository (see the update section below for more -details):: - - $ git config branch.autosetuprebase always - -By default when updating, Git will attempt to merge the remote changes and your -local changes. Ignoring what that sentence means, this is not an SVN-esque -model. Instead we want any local changes that we have to be applied on top of -any new remote changes. The 'branch.autosetuprebase' flag causes this to be done -automatically when ever one updates the local repository. - -Update ------- - -In SVN, one updates your local repository by running the update command:: - - $ SVN update - -In Git, instead of performing SVN update, one pulls from the remote repository:: - - $ git pull --rebase origin master - -This will pull any new remote commits into your local repository and then replay -your current local commits on top of those new commits. - -By default the '--rebase' flag is not necessary for the Swift repository because -it is configured to always rebase by setting the 'branch.autosetuprebase' flag -(see the section 'Repository Configuration (Enabling Default Rebasing)' above). - -Commit ------- - -In SVN, committing always means submitting changes to a remote repository. In -Git, committing refers to the process of first telling Git to track a change by -staging the change and then committing all staged changes into a change in the -local repository. One can have many such commits. Then when one is ready, one -pushes the new local changes to the remote repository. We go through these steps -in more detail below: - -In terms of replicating the SVN model, there are now two steps. In order to -commit changes one first stages a changed file using 'git add':: - - $ git add - -Then once all changes that you want to be apart of the commit have been staged, -a commit is created in the local repository via the 'commit' command:: - - $ git commit - -As a shortcut to commit /all/ changes to local files that are already being -tracked by Git to the local repository, you can use the '-a' command:: - - $ git commit -a - -In both of these cases, an editor will pop up to accept a commit message. To -specify a short commit message at the commandline, you can use the '-m' flag:: - - $ git commit -m 'My great commit message.' - -In order to see the diff of changes that have not been staged, run the command:: - - $ git diff - -To see all changes that have been staged, run the command:: - - $ git diff --staged - -To get a diff for a specific revision/path, perform the following command:: - - $ git diff - -In order to get a more concise view of the files that have staged and or -unstaged changes, run the command:: - - $ git status - -In order to restore a file from the last revision, one uses the checkout -command:: - - $ git checkout - -To restore a file to a specific revision, one must use a longer form of the -command:: - - $ git checkout -- - -To unstage a file, one uses the 'reset' command:: - - $ git reset - -This tells Git to reset '' in the staging area to the top of tree commit -(which in Git is called 'HEAD'). In order to correct a mistake, you can pass the -'amend' flag to Git:: - - $ git commit --amend - -This will cause all staged changes to be merged into 'HEAD'. Once one has made -all the relevant commits, in order to push the changes to the remote repository -the 'push' command is used:: - - $ git push origin master - -If a different committer has committed changes such that there are remote -commits that are not present locally, this will fail. In order to get around -this issue, perform:: - - $ git pull --rebase origin master - -in order to pull the new remote commits and replay your new commits on top. Then -try to push again. See the 'Checkout' section above how to configure the local -swift repository to always rebase allowing you to drop the '--rebase' flag. - -Revert ------- - -In SVN reverting a commit implies performing a reverse merge. In Git, this is no -longer true. Instead one now just uses the 'revert' command:: - - $ git revert - -This will cause Git to perform the reverse merge of that revision for you -against HEAD and bring up a message window for you to write a commit -message. This will be autofilled in with the title of the commit that is going -to be reverted and the revision number of that commit like so:: - - Revert "" - - This reverts commit . - -One can edit this message as one sees fit. Once this has been done, the revert -will become a normal commit in your repository like any other commit. Thus to -revert the commit in the remote repository, you need to perform a Git push:: - - $ git push origin master - -Browsing --------- - -This section explains how one can view Git changes. In order to view a history -of all changes on a branch to the beginning of time use the 'log' command:: - - $ git log - -This will for each commit show the following information:: - - commit - Author: - Date: - - - -To see history starting at a specific commit use the following form of a Git log -command:: - - $ git log - -To see a oneline summary that includes just the title of the commit and its -hash, pass the '--oneline' command:: - - $ git log --oneline - -It will not show you what was actually changed in each commit. In order to see -what was actually changed in a commit, use the command 'show':: - - $ git show - -This will show the aforementioned information shown by Git log, but additionally -will perform a diff against top of tree showing you the contents of the -change. To see the changes for a specific commit, pass the revision to Git -show:: - - $ git show diff --git a/docs/HighLevelSILOptimizations.md b/docs/HighLevelSILOptimizations.md new file mode 100644 index 0000000000000..ae82952f85a57 --- /dev/null +++ b/docs/HighLevelSILOptimizations.md @@ -0,0 +1,351 @@ +High-Level Optimizations in SIL +=============================== + +Abstract +-------- + +This document describes the high-level abstraction of built-in Swift +data structures in SIL that is used by the optimizer. You need to read +this document if you wish to understand the early stages of the Swift +optimizer or if you are working on one of the containers in the standard +library. + +Why do we need high-level optimizations? +---------------------------------------- + +Swift containers are implemented in the Swift standard library in Swift +code. Traditional compiler optimizations can remove some of the +redundancy that is found in high-level code, but not all of it. Without +knowledge of the Swift language the optimizer can't perform high-level +optimizations on the built-in containers. For example: + + Dict["a"] = 1 + Dict["a"] = 2 + +Any Swift developer could identify the redundancy in the code sample +above. Storing two values into the same key in the dictionary is +inefficient. However, optimizing compilers are unaware of the special +semantics that the Swift dictionary has and can't perform this +optimization. Traditional compilers would start optimizing this code by +inlining the subscript function call and try to analyze the sequence of +load/store instructions. This approach is not very effective because the +compiler has to be very conservative when optimizing general code with +pointers. + +On the other hand, compilers for high-level languages usually have +special bytecode instructions that allow them to perform high-level +optimizations. However, unlike high-level languages such as JavaScript +or Python, Swift containers are implemented in Swift itself. Moreover, +it is beneficial to be able to inline code from the container into the +user program and optimize them together, especially for code that uses +Generics. + +In order to perform both high-level optimizations, that are common in +high-level languages, and low-level optimizations we annotate parts of +the standard library and describe the semantics of a domain-specific +high-level operations on data types in the Swift standard library. + +Annotation of code in the standard library +------------------------------------------ + +We use the `@_semantics` attribute to annotate code in the standard +library. These annotations can be used by the high-level SIL optimizer +to perform domain-specific optimizations. + +This is an example of the `@_semantics` attribute: + + @public @_semantics("array.count") + func getCount() -> Int { + return _buffer.count + } + +In this example we annotate a member of the Swift array struct with the +tag `array.count`. This tag informs the optimizer that this method reads +the size of the array. + +The `@_semantics` attribute allows us to define "builtin" SIL-level +operations implemented in Swift code. In SIL code they are encoded as +apply instructions, but the optimizer can operate on them as atomic +instructions. The semantic annotations don't necessarily need to be on +public APIs. For example, the Array subscript operator may invoke two +operations in the semantic model. One for checking the bounds and +another one for accessing the elements. With this abstraction the +optimizer can remove the `checkSubscript` instruction and keep the +getElement instruction: + + @public subscript(index: Int) -> Element { + get { + checkSubscript(index) + return getElement(index) + } + + @_semantics("array.check_subscript") func checkSubscript(index: Int) { + ... + } + + @_semantics("array.get_element") func getElement(index: Int) -> Element { + return _buffer[index] + } + +Swift optimizations +------------------- + +The swift optimizer can access the information that is provided by the +`@_semantics` attribute to perform high-level optimizations. In the +early stages of the optimization pipeline the optimizer does not inline +functions with special semantics in order to allow the early high-level +optimization passes to operate on them. In the later stages of the +optimization pipeline the optimizer inlines functions with special +semantics to allow low-level optimizations. + +Annotated data structures in the standard library +------------------------------------------------- + +This section describes the semantic tags that are assigned to +data-structures in the standard library and the axioms that the +optimizer uses. + +### Cloning code from the standard library + +The Swift compiler can copy code from the standard library into the +application. This allows the optimizer to inline calls from stdlib and +improve the performance of code that uses common operators such as '++' +or basic containers such as Array. However, importing code from the +standard library can increase the binary size. Marking functions with +@\_semantics("stdlib\_binary\_only") will prevent the copying of the +marked function from the standard library into the user program. + +Notice that this annotation is similar to the resilient annotation that +will disallow the cloning of code into the user application. + +### Array + +The following semantic tags describe Array operations. The operations +are first described in terms of the Array "state". Relations between the +operations are formally defined below. 'Array' referes to the standard +library Array<T>, ContigousArray<T>, and ArraySlice<T> +data-structures. + +We consider the array state to consist of a set of disjoint elements and +a storage descriptor that encapsulates nonelement data such as the +element count and capacity. Operations that semantically write state are +always *control dependent*. A control dependent operation is one that +may only be executed on the control flow paths in which the operation +originally appeared, ignoring potential program exits. Generally, +operations that only read state are not control dependent. One exception +is `check_subscript` which is readonly but control dependent because it +may trap. Some operations are *guarded* by others. A guarded operation +can never be executed before its guard. + +array.init + +> Initialize an array with new storage. This currently applies to any +> initializer that does not get its storage from an argument. This +> semantically writes to every array element and the array's storage +> descriptor. `init` also implies the guarding semantics of +> `make_mutable`. It is not itself guarded by `make_mutable` and may act +> as a guard to other potentially mutating operations, such as +> `get_element_address`. + +array.uninitialized(count: Builtin.Word) -> (Array<T>, +Builtin.RawPointer) + +> Creates an array with the specified number of elements. It initializes +> the storage descriptor but not the array elements. The returned tuple +> contains the new array and a raw pointer to the element storage. The +> caller is responsible for writing the elements to the element storage. + +array.props.isCocoa/needsElementTypeCheck -> Bool + +: Reads storage descriptors properties + (isCocoa, needsElementTypeCheck). This is not control dependent + or guarded. The optimizer has semantic knowledge of the state + transfer those properties can not make: An array that is not + `isCocoa` can not transfer to `isCocoa`. An array that is not + `needsElementTypeCheck` can not transfer to `needsElementTypeCheck`. + +array.get\_element(index: Int) -> Element + +> Read an element from the array at the specified index. No other +> elements are read. The storage descriptor is not read. No state is +> written. This operation is not control dependent, but may be guarded +> by `check_subscript`. Any `check_subscript` may act as a guard, +> regardless of the index being checked [^1]. + +array.get\_element\_address(index: Int) -> +UnsafeMutablePointer<Element> + +> Get the address of an element of the array. No state is written. The +> storage descriptor is not read. The resulting pointer may be used to +> access elements in the array. This operation is not control dependent, +> but may be guarded by `check_subscript`. Any `check_subscript`, +> `make_mutable` or `mutate_unknown` may act as a guard. + +array.check\_subscript(index: Int) + +> Read the array count from the storage descriptor. Execute a `trap` if +> `index < array.startIndex || index >= array.endIndex`. No elements are +> read. No state is written. Despite being read only, this operation is +> control dependent. + +array.get\_count() -> Int + +> Read the array count (`array.endIndex - array.startIndex`) from the +> storage descriptor. No elements are read. No state is written. This is +> neither guarded nor control dependent. + +array.get\_capacity() -> Int + +> Read the array capacity from the storage descriptor. The semantics are +> identical to `get_count` except for the meaning of the return value. + +array.make\_mutable() + +> This operation guards mutating operations that don't already imply +> `make_mutable` semantics. (Currently, the only guarded operation is +> `get_element_address`.) `make_mutable` may create a copy of the array +> storage; however, semantically it neither reads nor writes the array +> state. It does not write state simply because the copy's state is +> identical to the original. It does not read state because no other +> Array operations can undo mutability--only code that retains a +> reference to the Array can do that. `make_mutable` does effectively +> need to be guarded by any SIL operation that may retain the array. +> Because `make_mutable` semantically does not read the array state, is +> idempotent, and has no control dependence, it can be executed safely +> on any array at any point. i.e. the optimizer can freely insert calls +> to make\_mutable. + +array.mutate\_unknown + +> This operation may mutate the array in any way, so it semantically +> writes to the entire array state and is naturally control dependent. +> `mutate_unknown` also implies the guarding semantics of +> `make_mutable`. It is not itself guarded by `make_mutable` and may act +> as a guard to other mutating operations, such as +> `get_element_address`. Combining semantics allows the flexibility in +> how the array copy is implemented in conjunction with implementing +> mutating functionality. This may be more efficient than cleanly +> isolating the copy and mutation code. + +To complete the semantics understood by the optimizer, we define these +relations: + +interferes-with + +> Given idempotent `OpA`, the sequence "`OpA, OpB, OpA`" is semantically +> equivalent to the sequence "`OpA, OpB`" *iff* `OpB` does not interfere +> with `OpA`. +> +> All array operations marked with semantics are idempotent as long as +> they call the same function with the same argument values, with the +> exception of `mutate_unknown`. + +guards + +> If `OpA` guards `OpB`, then the sequence of operations `OpA,OpB` must +> be preserved on any control flow path on which the sequence originally +> appears. + +An operation can only interfere-with or guard another if they may +operate on the same Array. `get_element_address` is abbreviated with +`get_elt_addr` in the table below. + + semantic op relation semantic ops + ------------------ ----------------- --------------------------------------------- + make\_mutable guards get\_element\_address + check\_subscript guards get\_element, get\_element\_address + make\_mutable interferes-with props.isCocoa/needsElementTypeCheck + get\_elt\_addr interferes-with get\_element, get\_element\_address, + props.isCocoa/needsElementTypeCheck + mutate\_unknown itereferes-with get\_element, check\_subscript, get\_count, + get\_capacity, get\_element\_address, + props.isCocoa/needsElementTypeCheck + +In addition to preserving these semantics, the optimizer must +conservatively handle any unknown access to the array object. For +example, if a SIL operation takes the address to any member of the +Array, any subsequent operations that may have visibility of that +address are considered to interfere with any array operations with +explicit semantics. + +### String + +string.concat(lhs: String, rhs: String) -> String + +> Performs concatenation of two strings. Operands are not mutated. This +> operation can be optimized away in case of both operands being string +> literals. In this case, it can be replaced by a string literal +> representing a concatenation of both operands. + +string.makeUTF8(start: RawPointer, byteSize: Word, isASCII: Int1) -> +String + +> Converts a built-in UTF8-encoded string literal into a string. + +string.makeUTF16(start: RawPointer, numberOfCodeUnits: Word) -> +String + +> Converts a built-in UTF16-encoded string literal into a string. + +### Dictionary + +TBD. + +### @effects attribute + +The @effects attribute describes how a function affects "the state of +the world". More practically how the optimizer can modify the program +based on information that is provided by the attribute. + +Usage: + +> @effects(readonly) func foo() { .. } + +The @effects attribute supports the following tags: + +readnone + +> function has no side effects and no dependencies on the state of the +> program. It always returns an identical result given identical inputs. +> Calls to readnone functions can be eliminated, reordered, and folded +> arbitrarily. + +readonly + +> function has no side effects, but is dependent on the global state of +> the program. Calls to readonly functions can be eliminated, but cannot +> be reordered or folded in a way that would move calls to the readnone +> function across side effects. + +readwrite + +> function has side effects and the optimizer can't assume anything. + +### Optimize semantics attribute + +The optimize attribute adds function-specific directives to the +optimizer. + +The optimize attribute supports the following tags: + +sil.never + +> The sil optimizer should not optimize this function. + +> Example: @\_semantics("optimize.sil.never") func miscompile() { ... } + +### Availability checks + +The availability attribute is used for functions which implement the +`if #available` guards. + +The availability attribute supports the following tags: + +availability.osversion(major: Builtin.Word, minor: Builtin.Word, patch: +Builtin.Word) -> Builtin.Int1 + +> Returns true if the OS version matches the parameters. + +[^1]: Any check\_subscript(N) may act as a guard for + `get_element(i)/get_element_address(i)` as long as it can be shown + that `N >= i`. diff --git a/docs/HighLevelSILOptimizations.rst b/docs/HighLevelSILOptimizations.rst deleted file mode 100644 index fa2132859593d..0000000000000 --- a/docs/HighLevelSILOptimizations.rst +++ /dev/null @@ -1,361 +0,0 @@ -:orphan: - -.. _HighLevelSILOptimizations: - -High-Level Optimizations in SIL -=============================== - -.. contents:: - -Abstract --------- - -This document describes the high-level abstraction of built-in Swift -data structures in SIL that is used by the optimizer. You need to read -this document if you wish to understand the early stages of the Swift -optimizer or if you are working on one of the containers in the -standard library. - - -Why do we need high-level optimizations? ------------------------------------------ - -Swift containers are implemented in the Swift standard library in Swift code. -Traditional compiler optimizations can remove some of the redundancy that is -found in high-level code, but not all of it. Without knowledge of the Swift -language the optimizer can't perform high-level optimizations on the built-in -containers. For example:: - - Dict["a"] = 1 - Dict["a"] = 2 - -Any Swift developer could identify the redundancy in the code sample above. -Storing two values into the same key in the dictionary is inefficient. -However, optimizing compilers are unaware of the special semantics that the -Swift dictionary has and can't perform this optimization. Traditional -compilers would start optimizing this code by inlining the subscript -function call and try to analyze the sequence of load/store instructions. -This approach is not very effective because the compiler has to be very -conservative when optimizing general code with pointers. - -On the other hand, compilers for high-level languages usually have special -bytecode instructions that allow them to perform high-level optimizations. -However, unlike high-level languages such as JavaScript or Python, Swift -containers are implemented in Swift itself. Moreover, it is beneficial to -be able to inline code from the container into the user program and optimize -them together, especially for code that uses Generics. - -In order to perform both high-level optimizations, that are common in -high-level languages, and low-level optimizations we annotate parts of the -standard library and describe the semantics of a domain-specific high-level -operations on data types in the Swift standard library. - -Annotation of code in the standard library ------------------------------------------- - -We use the ``@_semantics`` attribute to annotate code in the standard library. -These annotations can be used by the high-level SIL optimizer to perform -domain-specific optimizations. - -This is an example of the ``@_semantics`` attribute:: - - @public @_semantics("array.count") - func getCount() -> Int { - return _buffer.count - } - -In this example we annotate a member of the Swift array struct with the tag -``array.count``. This tag informs the optimizer that this method reads the -size of the array. - - -The ``@_semantics`` attribute allows us to define "builtin" SIL-level -operations implemented in Swift code. In SIL code they are encoded as -apply instructions, but the optimizer can operate on them as atomic -instructions. The semantic annotations don't necessarily need to be on -public APIs. For example, the Array subscript operator may invoke two -operations in the semantic model. One for checking the bounds and -another one for accessing the elements. With this abstraction the -optimizer can remove the ``checkSubscript`` instruction and keep the -getElement instruction:: - - @public subscript(index: Int) -> Element { - get { - checkSubscript(index) - return getElement(index) - } - - @_semantics("array.check_subscript") func checkSubscript(index: Int) { - ... - } - - @_semantics("array.get_element") func getElement(index: Int) -> Element { - return _buffer[index] - } - - -Swift optimizations -------------------- -The swift optimizer can access the information that is provided by the -``@_semantics`` attribute to perform high-level optimizations. In the early -stages of the optimization pipeline the optimizer does not inline functions -with special semantics in order to allow the early high-level optimization -passes to operate on them. In the later stages of the optimization pipeline -the optimizer inlines functions with special semantics to allow low-level -optimizations. - - -Annotated data structures in the standard library -------------------------------------------------- - -This section describes the semantic tags that are assigned to data-structures -in the standard library and the axioms that the optimizer uses. - - -Cloning code from the standard library -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The Swift compiler can copy code from the standard library into the -application. This allows the optimizer to inline calls from stdlib and improve -the performance of code that uses common operators such as '++' or basic -containers such as Array. However, importing code from the standard library can -increase the binary size. Marking functions with @_semantics("stdlib_binary_only") -will prevent the copying of the marked function from the standard library into the -user program. - -Notice that this annotation is similar to the resilient annotation that will -disallow the cloning of code into the user application. - -Array -~~~~~ - -The following semantic tags describe Array operations. The operations -are first described in terms of the Array "state". Relations between the -operations are formally defined below. 'Array' referes to the standard library -Array, ContigousArray, and ArraySlice data-structures. - -We consider the array state to consist of a set of disjoint elements -and a storage descriptor that encapsulates nonelement data such as the -element count and capacity. Operations that semantically write state -are always *control dependent*. A control dependent operation is one -that may only be executed on the control flow paths in which the -operation originally appeared, ignoring potential program -exits. Generally, operations that only read state are not control -dependent. One exception is ``check_subscript`` which is readonly but -control dependent because it may trap. Some operations are *guarded* -by others. A guarded operation can never be executed before its -guard. - -array.init - - Initialize an array with new storage. This currently applies to any - initializer that does not get its storage from an argument. This - semantically writes to every array element and the array's storage - descriptor. ``init`` also implies the guarding semantics of - ``make_mutable``. It is not itself guarded by ``make_mutable`` and - may act as a guard to other potentially mutating operations, such as - ``get_element_address``. - -array.uninitialized(count: Builtin.Word) -> (Array, Builtin.RawPointer) - - Creates an array with the specified number of elements. It initializes - the storage descriptor but not the array elements. The returned tuple - contains the new array and a raw pointer to the element storage. - The caller is responsible for writing the elements to the element storage. - -array.props.isCocoa/needsElementTypeCheck -> Bool - Reads storage descriptors properties (isCocoa, needsElementTypeCheck). - This is not control dependent or guarded. The optimizer has - semantic knowledge of the state transfer those properties can not make: - An array that is not ``isCocoa`` can not transfer to ``isCocoa``. - An array that is not ``needsElementTypeCheck`` can not transfer to - ``needsElementTypeCheck``. - -array.get_element(index: Int) -> Element - - Read an element from the array at the specified index. No other - elements are read. The storage descriptor is not read. No state is - written. This operation is not control dependent, but may be - guarded by ``check_subscript``. Any ``check_subscript`` may act as a - guard, regardless of the index being checked [#f1]_. - -array.get_element_address(index: Int) -> UnsafeMutablePointer - - Get the address of an element of the array. No state is written. The storage - descriptor is not read. The resulting pointer may be used to access elements - in the array. This operation is not control dependent, but may be guarded by - ``check_subscript``. Any ``check_subscript``, ``make_mutable`` or - ``mutate_unknown`` may act as a guard. - -array.check_subscript(index: Int) - - Read the array count from the storage descriptor. Execute a ``trap`` - if ``index < array.startIndex || index >= array.endIndex``. No elements are - read. No state is written. Despite being read only, this operation is control - dependent. - -array.get_count() -> Int - - Read the array count (``array.endIndex - array.startIndex``) from the storage - descriptor. No elements are read. No state is written. This is neither guarded - nor control dependent. - -array.get_capacity() -> Int - - Read the array capacity from the storage descriptor. The semantics - are identical to ``get_count`` except for the meaning of the return value. - -array.make_mutable() - - This operation guards mutating operations that don't already imply - ``make_mutable`` semantics. (Currently, the only guarded operation - is ``get_element_address``.) ``make_mutable`` may create a copy of the array - storage; however, semantically it neither reads nor writes the array - state. It does not write state simply because the copy's state is - identical to the original. It does not read state because no other - Array operations can undo mutability--only code that retains a - reference to the Array can do that. ``make_mutable`` does - effectively need to be guarded by any SIL operation that may retain - the array. Because ``make_mutable`` semantically does not read the - array state, is idempotent, and has no control dependence, it can be - executed safely on any array at any point. i.e. the optimizer can - freely insert calls to make_mutable. - -array.mutate_unknown - - This operation may mutate the array in any way, so it semantically - writes to the entire array state and is naturally control - dependent. ``mutate_unknown`` also implies the guarding semantics of - ``make_mutable``. It is not itself guarded by ``make_mutable`` and - may act as a guard to other mutating operations, such as - ``get_element_address``. Combining semantics allows the flexibility in how - the array copy is implemented in conjunction with implementing - mutating functionality. This may be more efficient than cleanly - isolating the copy and mutation code. - -To complete the semantics understood by the optimizer, we define these relations: - -interferes-with - - Given idempotent ``OpA``, the sequence "``OpA, OpB, OpA``" is - semantically equivalent to the sequence "``OpA, OpB``" *iff* ``OpB`` - does not interfere with ``OpA``. - - All array operations marked with semantics are idempotent as long as - they call the same function with the same argument values, with the - exception of ``mutate_unknown``. - -guards - - If ``OpA`` guards ``OpB``, then the sequence of operations - ``OpA,OpB`` must be preserved on any control flow path on which the - sequence originally appears. - -An operation can only interfere-with or guard another if they may operate on the same Array. -``get_element_address`` is abbreviated with ``get_elt_addr`` in the table below. - -================ =============== ========================================== -semantic op relation semantic ops -================ =============== ========================================== -make_mutable guards get_element_address -check_subscript guards get_element, get_element_address -make_mutable interferes-with props.isCocoa/needsElementTypeCheck -get_elt_addr interferes-with get_element, get_element_address, - props.isCocoa/needsElementTypeCheck -mutate_unknown itereferes-with get_element, check_subscript, get_count, - get_capacity, get_element_address, - props.isCocoa/needsElementTypeCheck -================ =============== ========================================== - -.. [#f1] Any check_subscript(N) may act as a guard for - ``get_element(i)/get_element_address(i)`` as long as it can be - shown that ``N >= i``. - -In addition to preserving these semantics, the optimizer must -conservatively handle any unknown access to the array object. For -example, if a SIL operation takes the address to any member of the -Array, any subsequent operations that may have visibility of that -address are considered to interfere with any array operations with -explicit semantics. - -String -~~~~~~ - -string.concat(lhs: String, rhs: String) -> String - - Performs concatenation of two strings. Operands are not mutated. - This operation can be optimized away in case of both operands - being string literals. In this case, it can be replaced by - a string literal representing a concatenation of both operands. - -string.makeUTF8(start: RawPointer, byteSize: Word, isASCII: Int1) -> String - - Converts a built-in UTF8-encoded string literal into a string. - -string.makeUTF16(start: RawPointer, numberOfCodeUnits: Word) -> String - - Converts a built-in UTF16-encoded string literal into a string. - -Dictionary -~~~~~~~~~~ -TBD. - -@effects attribute -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The @effects attribute describes how a function affects "the state of the world". -More practically how the optimizer can modify the program based on information -that is provided by the attribute. - -Usage: - - @effects(readonly) func foo() { .. } - - -The @effects attribute supports the following tags: - -readnone - - function has no side effects and no dependencies on the state of - the program. It always returns an identical result given - identical inputs. Calls to readnone functions can be eliminated, - reordered, and folded arbitrarily. - -readonly - - function has no side effects, but is dependent on the global - state of the program. Calls to readonly functions can be - eliminated, but cannot be reordered or folded in a way that would - move calls to the readnone function across side effects. - -readwrite - - function has side effects and the optimizer can't assume anything. - -Optimize semantics attribute -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The optimize attribute adds function-specific directives to the optimizer. - -The optimize attribute supports the following tags: - -sil.never - - The sil optimizer should not optimize this function. - - Example: - @_semantics("optimize.sil.never") - func miscompile() { ... } - -Availability checks -~~~~~~~~~~~~~~~~~~~ - -The availability attribute is used for functions which implement the ``if #available`` -guards. - -The availability attribute supports the following tags: - -availability.osversion(major: Builtin.Word, minor: Builtin.Word, patch: Builtin.Word) -> Builtin.Int1 - - Returns true if the OS version matches the parameters. - diff --git a/docs/Import.md b/docs/Import.md new file mode 100644 index 0000000000000..4494d41ff2763 --- /dev/null +++ b/docs/Import.md @@ -0,0 +1,169 @@ +> **warning** +> +> This document represents an early proposal for `import` syntax and + +> has not been kept up to date. + +IMPORT SYNTAX +============= + + import-decl ::= 'import' import-item-list + import-item-list ::= import-item (',' import-item)* + + import-item ::= import-kind? identifier-path + import-item ::= identifier-path '.' '(' import-item-list ')' + + import-kind ::= 'module' + + import-kind ::= 'class' + import-kind ::= 'enum' + import-kind ::= 'func' + import-kind ::= 'protocol' + import-kind ::= 'struct' + import-kind ::= 'typealias' + import-kind ::= 'var' + // ... + +`import` makes declarations exported from another module available +inside the current module. Imports are not reexported by default. + +Importing Modules +----------------- + +In its simplest form, `import` gives the qualified name of a module and +imports all exported symbols from the module, as well as the module name +itself for qualified lookup: + + import Cocoa + + // Reference the NSArray type from Cocoa + var a1 : NSArray + // Same, but qualified + var a2 : Cocoa.NSArray + +In this form, the qualified name *must* refer to a module: + + // Import the Cocoa.NSWindow module, *not* the NSWindow class from inside + // Cocoa + import Cocoa.NSWindow + + // Reference the NSWindow type from Cocoa.NSWindow + var w1 : NSWindow + // Same, but qualified + var w2 : Cocoa.NSWindow.NSWindow + +Multiple modules may appear in a comma-separated list: + + import Foundation, iAd, CoreGraphics + +As a shorthand, multiple submodules with a common parent module may be +listed in parens under the parent module: + + import OpenGL.(GL3, GL3.Ext) + +Importing Individual Declarations +--------------------------------- + +Instead of importing the entire contents of a module, individual +declarations may be imported. This is done by naming the kind of +declaration being imported before the qualified name, such as `func`, +`var`, or `class`. The module name is still imported for qualified +lookup of other symbols: + + // Import only the Cocoa.NSWindow class + import class Cocoa.NSWindow + + var w1 : NSWindow + var title : Cocoa.NSString + +As with modules, multiple declarations may be imported in a +comma-separated list, or imported out of a common parent module with a +parenthesized list: + + import func OpenGL.GL3.glDrawArrays, func OpenGL.GL3.Ext.glTextureRangeAPPLE + // Equivalent + import OpenGL.GL3.(func glDrawArrays, func Ext.glTextureRangeAPPLE) + +RESOLVING NAME CLASHES +====================== + +Module imports +-------------- + +Because the local names introduced by a whole-module import are +implicit, a name clash between imported modules is not an error unless a +clashing name is actually used without qualification: + + import abcde // abcde exports A, B, C, D, E + import aeiou // aeiou exports A, E, I, O, U + + var b : B // OK, references abcde.B + var i : I // OK, references aeiou.I + var e : E // Error, ambiguous + var e : abcde.E // OK, qualified reference to abcde.E + +Conflicts are resolved in favor of individually imported or locally +defined declarations when available: + + import abcde // abcde exports A, B, C, D, E + import aeiou // aeiou exports A, E, I, O, U + import class asdf.A // explicitly import A from some other module + import class abcde.E // explicitly import E from abcde + + class U { } // Local class shadows whole-module import + + var a : A // OK, references asdf.A + var e : E // OK, references abcde.E + var u : U // OK, references local U + +Declaration imports +------------------- + +Individual declaration imports shadow whole-module imports, as described +above. If two declarations with the same name are individually imported +from different modules, references to either import must be qualified: + + import class abcde.E + import class aeiou.E + + var e : E // Error, ambiguous + var e1 : abcde.E // OK + +A local definition with the same name as an explicitly imported symbol +shadows the unqualified import: + + import class abcde.E + + class E { } + + var e : E // Refers to local E + var e : abcde.E // Refers to abcde.E + +Module names +------------ + +FIXME: What is a good rule here? This sucks. + +If a module name clashes with a local definition or imported +declaration, the declaration is favored in name lookup. If a member +lookup into the declaration fails, we fall back to qualified lookup into +the module: + + import Foo // exports bas + + class Foo { + class func bar() + } + + Foo.bar() // bar method from Foo class + Foo.bas() // bas method from Foo module + +FUTURE EXTENSIONS +================= + +In the future, we should allow the import declaration to provide an +alias for the imported module or declaration: + + import C = Cocoa + import NSW = class Cocoa.NSWindow + import Cocoa.(NSW = class NSWindow, NSV = class NSView) diff --git a/docs/Import.rst b/docs/Import.rst deleted file mode 100644 index 98c197e4b9481..0000000000000 --- a/docs/Import.rst +++ /dev/null @@ -1,167 +0,0 @@ -:orphan: - -.. warning:: This document represents an early proposal for ``import`` syntax and - has not been kept up to date. - -IMPORT SYNTAX -============= -:: - - import-decl ::= 'import' import-item-list - import-item-list ::= import-item (',' import-item)* - - import-item ::= import-kind? identifier-path - import-item ::= identifier-path '.' '(' import-item-list ')' - - import-kind ::= 'module' - - import-kind ::= 'class' - import-kind ::= 'enum' - import-kind ::= 'func' - import-kind ::= 'protocol' - import-kind ::= 'struct' - import-kind ::= 'typealias' - import-kind ::= 'var' - // ... - -``import`` makes declarations exported from another module available inside -the current module. Imports are not reexported by default. - -Importing Modules ------------------ - -In its simplest form, ``import`` gives the qualified name of a module and -imports all exported symbols from the module, as well as the module name itself -for qualified lookup:: - - import Cocoa - - // Reference the NSArray type from Cocoa - var a1 : NSArray - // Same, but qualified - var a2 : Cocoa.NSArray - -In this form, the qualified name *must* refer to a module:: - - // Import the Cocoa.NSWindow module, *not* the NSWindow class from inside - // Cocoa - import Cocoa.NSWindow - - // Reference the NSWindow type from Cocoa.NSWindow - var w1 : NSWindow - // Same, but qualified - var w2 : Cocoa.NSWindow.NSWindow - -Multiple modules may appear in a comma-separated list:: - - import Foundation, iAd, CoreGraphics - -As a shorthand, multiple submodules with a common parent module may be listed -in parens under the parent module:: - - import OpenGL.(GL3, GL3.Ext) - -Importing Individual Declarations ---------------------------------- - -Instead of importing the entire contents of a module, individual declarations -may be imported. This is done by naming the kind of declaration being imported -before the qualified name, such as ``func``, ``var``, or ``class``. The module -name is still imported for qualified lookup of other symbols:: - - // Import only the Cocoa.NSWindow class - import class Cocoa.NSWindow - - var w1 : NSWindow - var title : Cocoa.NSString - -As with modules, multiple declarations may be imported in a comma-separated -list, or imported out of a common parent module with a parenthesized list:: - - import func OpenGL.GL3.glDrawArrays, func OpenGL.GL3.Ext.glTextureRangeAPPLE - // Equivalent - import OpenGL.GL3.(func glDrawArrays, func Ext.glTextureRangeAPPLE) - -RESOLVING NAME CLASHES -====================== - -Module imports --------------- - -Because the local names introduced by a whole-module import are implicit, -a name clash between imported modules is not an error unless a clashing name is -actually used without qualification:: - - import abcde // abcde exports A, B, C, D, E - import aeiou // aeiou exports A, E, I, O, U - - var b : B // OK, references abcde.B - var i : I // OK, references aeiou.I - var e : E // Error, ambiguous - var e : abcde.E // OK, qualified reference to abcde.E - -Conflicts are resolved in favor of individually imported or -locally defined declarations when available:: - - import abcde // abcde exports A, B, C, D, E - import aeiou // aeiou exports A, E, I, O, U - import class asdf.A // explicitly import A from some other module - import class abcde.E // explicitly import E from abcde - - class U { } // Local class shadows whole-module import - - var a : A // OK, references asdf.A - var e : E // OK, references abcde.E - var u : U // OK, references local U - -Declaration imports -------------------- - -Individual declaration imports shadow whole-module imports, as described above. -If two declarations with the same name are individually imported from different -modules, references to either import must be qualified:: - - import class abcde.E - import class aeiou.E - - var e : E // Error, ambiguous - var e1 : abcde.E // OK - -A local definition with the same name as an explicitly imported symbol -shadows the unqualified import:: - - import class abcde.E - - class E { } - - var e : E // Refers to local E - var e : abcde.E // Refers to abcde.E - -Module names ------------- - -FIXME: What is a good rule here? This sucks. - -If a module name clashes with a local definition or imported declaration, the -declaration is favored in name lookup. If a member lookup into the declaration -fails, we fall back to qualified lookup into the module:: - - import Foo // exports bas - - class Foo { - class func bar() - } - - Foo.bar() // bar method from Foo class - Foo.bas() // bas method from Foo module - -FUTURE EXTENSIONS -================= - -In the future, we should allow the import declaration to provide an alias -for the imported module or declaration:: - - import C = Cocoa - import NSW = class Cocoa.NSWindow - import Cocoa.(NSW = class NSWindow, NSV = class NSView) - diff --git a/docs/IndexInvalidation.md b/docs/IndexInvalidation.md new file mode 100644 index 0000000000000..9c925bb5d5f76 --- /dev/null +++ b/docs/IndexInvalidation.md @@ -0,0 +1,174 @@ +Index Invalidation Rules in the Swift Standard Library +====================================================== + +Points to consider +------------------ + +(1) Collections can be implemented as value types or a reference types. +(2) Copying an instance of a value type, or copying a reference has + well-defined semantics built into the language and is not + controllable by the user code. + + Consequence: value-typed collections in Swift have to use + copy-on-write for data stored out-of-line in + reference-typed buffers. + +(3) We want to be able to pass/return a Collection along with its + indices in a safe manner. + + In Swift, unlike C++, indices are not sufficient to access + collection data; one needs an index and a collection. Thus, merely + passing a collection by value to a function should not + invalidate indices. + +General principles +------------------ + +In C++, validity of an iterator is a property of the iterator itself, +since iterators can be dereferenced to access collection elements. + +In Swift, in order to access a collection element designated by an +index, subscript operator is applied to the collection, `C[I]`. Thus, +index is valid or not only in context of a certain collection instance +at a certain point of program execution. A given index can be valid for +zero, one or more than one collection instance at the same time. + +An index that is valid for a certain collection designates an element of +that collection or represents a one-past-end index. + +Operations that access collection elements require valid indexes (this +includes accessing using the subscript operator, slicing, swapping +elements, removing elements etc.) + +Using an invalid index to access elements of a collection leads to +unspecified memory-safe behavior. (Possibilities include trapping, +performing the operation on an arbitrary element of this or any other +collection etc.) Concrete collection types can specify behavior; +implementations are advised to perform a trap. + +An arbitrary index instance is not valid for an arbitrary collection +instance. + +The following points apply to all collections, defined in the library or +by the user: + +(1) Indices obtained from a collection `C` via `C.startIndex`, + `C.endIndex` and other collection-specific APIs returning indices, + are valid for `C`. +(2) If an index `I` is valid for a collection `C`, a copy of `I` is + valid for `C`. +(3) If an index `I` is valid for a collection `C`, indices obtained from + `I` via `I.successor()`, `I.predecessor()`, and other index-specific + APIs, are valid for `C`. FIXME: disallow + startIndex.predecessor(), endIndex.successor() +(4) **Indices of collections and slices freely interoperate.** + + If an index `I` is valid for a collection `C`, it is also valid for + slices of `C`, provided that `I` was in the bounds that were passed + to the slicing subscript. + + If an index `I` is valid for a slice obtained from a collection `C`, + it is also valid for `C` itself. + +(5) If an index `I` is valid for a collection `C`, it is also valid for + a copy of `C`. +(6) If an index `I` is valid for a collection `C`, it continues to be + valid after a call to a non-mutating method on `C`. +(7) Calling a non-mutating method on a collection instance does not + invalidate any indexes. +(8) Indices behave as if they are composites of offsets in the + underlying data structure. For example: + + - an index into a set backed by a hash table with open addressing + is the number of the bucket where the element is stored; + - an index into a collection backed by a tree is a sequence of + integers that describe the path from the root of the tree to the + leaf node; + - an index into a lazy flatMap collection consists of a pair of + indices, an index into the base collection that is being mapped, + and the index into the result of mapping the element designated + by the first index. + + This rule does not imply that indices should be cheap to convert to + actual integers. The offsets for consecutive elements could be + non-consecutive (e.g., in a hash table with open addressing), or + consist of multiple offsets so that the conversion to an integer is + non-trivial (e.g., in a tree). + + Note that this rule, like all other rules, is an "as if" rule. As + long as the resulting semantics match what the rules dictate, the + actual implementation can be anything. + + Rationale and discussion: + + - This rule is mostly motivated by its consequences, in + particular, being able to mutate an element of a collection + without changing the collection's structure, and, thus, without + invalidating indices. + - Replacing a collection element has runtime complexity O(1) and + is not considered a structural mutation. Therefore, there seems + to be no reason for a collection model would need to invalidate + indices from the implementation point of view. + - Iterating over a collection and performing mutations in place is + a common pattern that Swift's collection library needs + to support. If replacing individual collection elements would + invalidate indices, many common algorithms (like sorting) + wouldn't be implementable directly with indices; the code would + need to maintain its own shadow indices, for example, plain + integers, that are not invalidated by mutations. + +Consequences: + +- The setter of `MutableCollection.subscript(_: Index)` does not + invalidate any indices. Indices are composites of offsets, so + replacing the value does not change the shape of the data structure + and preserves offsets. +- A value type mutable linked list can not conform to + `MutableCollectionType`. An index for a linked list has to be + implemented as a pointer to the list node to provide O(1) + element access. Mutating an element of a non-uniquely referenced + linked list will create a copy of the nodes that comprise the list. + Indices obtained before the copy was made would point to the old + nodes and wouldn't be valid for the copy of the list. + + It is still valid to have a value type linked list conform to + `CollectionType`, or to have a reference type mutable linked list + conform to `MutableCollection`. + +The following points apply to all collections by default, but specific +collection implementations can be less strict: + +(1) A call to a mutating method on a collection instance, except the + setter of `MutableCollection.subscript(_: Index)`, invalidates all + indices for that collection instance. + +Consequences: + +- Passing a collection as an `inout` argument invalidates all indexes + for that collection instance, unless the function explicitly + documents stronger guarantees. (The function can call mutating + methods on an `inout` argument or completely replace it.) + - `Swift.swap()` does not invalidate any indexes. + +Additional guarantees for `Swift.Array`, `Swift.ContiguousArray`, `Swift.ArraySlice` +------------------------------------------------------------------------------------ + +**Valid array indexes can be created without using Array APIs.** Array +indexes are plain integers. Integers that are dynamically in the range +`0.. **warning** +> +> **This document is still in draft stages.** Large additions and + +> restructuring are still planned, including: +> +> - A summary for each declaration kind what changes +> are binary-compatible. +> - A proper definition for "versioned entity". +> - Several possible versioned attribute syntaxes, instead of just +> this one. +> - A discussion of back-dating, and how it usually is not allowed. +> - A brief discussion of the implementation issues for fixed-layout +> value types with resilient members, and with non-public members. +> - A revisal of the discussion on fixed-layout classes. +> - A brief discussion of "deployment files", which represent +> distribution groupings that are themselves versioned. (For +> example, OS X 10.10.3 contains Foundation version 1153.20.) +> Deployment files are likely to provide a concrete implementation +> of "resilience domains". +> - A way to specify "minimum deployment libraries", like today's +> minimum deployment targets. + +Introduction +============ + +This model is intended to serve library designers whose libraries will +evolve over time. Such libraries must be both +[backwards-compatible](#backwards-compatible), meaning that existing clients +should continue to work even when the library is updated, and +[forwards-compatible](#forwards-compatible), meaning that future clients will be +able run using the current version of the library. In simple terms: + +- Last year's apps should work with this year's library. +- Next year's apps should work with this year's library. + +This document will frequently refer to a *library* which vends public +APIs, and a single *client* that uses them. The same principles apply +even when multiple libraries and multiple clients are involved. + +This model is not of interest to libraries that are bundled with their +clients (distribution via source, static library, or embedded/sandboxed +dynamic library). Because a client always uses a particular version of +such a library, there is no need to worry about backwards- or +forwards-compatibility. Just as developers with a single app target are +not forced to think about access control, anyone writing a bundled +library should not be required to use any of the annotations described +below in order to achieve full performance. + +The term "resilience" comes from the occasional use of "fragile" to +describe certain constructs that have very strict binary compatibility +rules. For example, a client's use of a C struct is "fragile" in that if +the library changes the fields in the struct, the client's use will +"break". In Swift, changing the fields in a struct will not +automatically cause problems for existing clients, so we say the struct +is "resilient". + +Using Versioned API +=================== + +References to a versioned API must always be guarded with the +appropriate availability checks. This means that any client entities +that rely on having certain APIs from a library must themselves be +restricted to contexts in which those APIs are available. This is +accomplished using `@available` as well, by specifying the name of the +client library along with the required version: + + // Client code + @available(Magician 1.5) + class CrystalBallView : MagicView { … } + +Library versions can also be checked dynamically using `#available`, +allowing for fallback behavior when the requested library version is not +present: + + func scareMySiblings() { + if #available(Magician 1.2) { + conjureDemons() + } else { + print("BOO!!") + } + } + +> **note** +> +> Possible implementations include generating a hidden symbol into a +> library, or putting the version number in some kind of metadata, like +> the Info.plist in a framework bundle on Darwin platforms. + +This is essentially the same model as the availability checking released +in Swift 2.0, but generalized for checking library versions instead of +just OS versions. + +Publishing Versioned API +======================== + +A library's API is already marked with the `public` attribute. +Versioning information can be added to any `public` entity with the +`@available` attribute, this time specifying *only* a version number. +This declares when the entity was first exposed publicly in the current +module. + + @available(1.2) + public func conjureDemons() + +> **TODO** +> +> Should this go on `public` instead? How does this play with SPI +> <rdar://problem/18844229>? + +Using the same attribute for both publishing and using versioned APIs +helps tie the feature together and enforces a consistent set of rules. +The one difference is that code within a library may always use all +other entities declared within the library (barring their own +availability checks), since the entire library is shipped as a unit. +That is, even if a particular API was introduced in v1.0, its +(non-public) implementation may refer to APIs introduced in later +versions. + +Swift libraries are strongly encouraged to use [semantic +versioning](http://semver.org), but this is not enforced by the +language. + +Some `internal` entities may also use `@available`. See [Pinning](#pinning) +below. + +Giving Up Flexibility +===================== + +Fixed-layout Structs +-------------------- + +By default, a library owner may add members to a public struct between +releases without breaking binary compatibility. This requires a certain +amount of care and indirection when dealing with values of struct type +to account for the struct's size and non-[trivial](#trivial) fields not being known +in advance, which of course has performance implications. + +To opt out of this flexibility, a struct may be marked `@fixed_layout`. +This promises that no stored properties will be added to or removed from +the struct, even `private` or `internal` ones. Methods and computed +properties may still be added to the struct. + +The `@fixed_layout` attribute takes a version number, just like +`@available`. This is so that clients can deploy against older versions +of the library, which may have a different layout for the struct. (In +this case the client must manipulate the struct as if the +`@fixed_layout` attribute were absent.) + +> **TODO** +> +> There's a benefit to knowing that a struct was `@fixed_layout` since +> it was first made available. How should that be spelled? + +### Fixed-layout Classes? + +There is some benefit to knowing that a class has a fixed layout---that +is, that the stored properties of the class and all its superclasses are +guaranteed not to change in future versions of a library. This would, +for example, allow the class's memory to be allocated on the stack, as +long as it can be proven that no references to the class escape. +However, such a constraint is unlikely to be provable in practice from +outside the class's own module, where its primary operations are +opaquely defined. Thus, until a tangible benefit has been demonstrated, +the `@fixed_layout` attribute will not apply to classes. + +(Another benefit would be to simplify the calculations needed for the +offsets of stored properties within classes. However, it's unclear that +this would have any significant benefit, particularly when most public +properties are manipulated through their accessors.) + +Closed Enums +------------ + +By default, a library owner may add new cases to a public enum between +releases without breaking binary compatibility. As with structs, this +results in a fair amount of indirection when dealing with enum values, +in order to potentially accommodate new values. + +> **note** +> +> If an enum value has a known case, or can be proven to belong to a set +> of known cases, the compiler is of course free to use a more efficient +> representation for the value, just as it may discard fields of structs +> that are provably never accessed. + +A library owner may opt out of this flexibility by marking the enum as +`@closed`. A "closed" enum may not have any `private` or `internal` +cases and may not add new cases in the future. This guarantees to +clients that the enum cases are exhaustive. + +> **note** +> +> Were a "closed" enum allowed to have non-public cases, clients of the +> library would still have to treat the enum as opaque and would still +> have to be able to handle unknown cases in their `switch` statements. + +The `@closed` attribute takes a version number, just like `@available`. +This is so that clients can deploy against older versions of the +library, which may have non-public cases in the enum. (In this case the +client must manipulate the enum as if the `@closed` attribute were +absent.) + +Even for default "open" enums, adding new cases should not be done +lightly. Any clients attempting to do an exhaustive switch over all enum +cases will likely not handle new cases well. + +> **note** +> +> One possibility would be a way to map new cases to older ones on older +> clients. This would only be useful for certain kinds of enums, though, +> and adds a lot of additional complexity, all of which would be tied up +> in versions. Our generalized switch patterns probably make it hard to +> nail down the behavior here. + +Inlineable Functions +-------------------- + +Functions are a very common example of resilience: the function's +declaration is published as API, but its body may change between library +versions as long as it upholds the same semantic contracts. This applies +to other function-like constructs as well: initializers, accessors, and +deinitializers. + +However, sometimes it is useful to provide the body to clients as well. +There are a few common reasons for this: + +- The function only performs simple operations, and so inlining it + will both save the overhead of a cross-library function call and + allow further optimization of callers. +- The function accesses a fixed-layout struct with non-public members; + this allows the library author to preserve invariants while still + allowing efficient access to the struct. + +A public function marked with the `@inlineable` attribute makes its body +available to clients as part of the module's public interface. The +`@inlineable` attribute takes a version number, just like `@available`; +clients may not assume that the body of the function is suitable when +deploying against older versions of the library. + +Clients are not required to inline a function marked `@inlineable`. + +> **note** +> +> It is legal to change the implementation of an inlineable function in +> the next release of the library. However, any such change must be made +> with the understanding that it may or may not affect existing clients. + +### Restrictions + +Because the body of an inlineable function (or method, accessor, +initializer, or deinitializer) may be inlined into another module, it +must not make any assumptions that rely on knowledge of the current +module. Here is a trivial example: + + public struct Point2D { + var x, y: Double + public init(x: Double, y: Double) { … } + } + + extension Point2D { + @inlineable public func distanceTo(other: Point2D) -> Double { + let deltaX = self.x - other.x + let deltaY = self.y - other.y + return sqrt(deltaX*deltaX + deltaY*deltaY) + } + } + +As written, this `distanceTo` method is not safe to inline. The next +release of the library could very well replace the implementation of +`Point2D` with a polar representation: + + public struct Point2D { + var r, theta: Double + public init(x: Double, y: Double) { … } + } + +and the `x` and `y` properties have now disappeared. To avoid this, we +have the following restrictions on the bodies of inlineable functions: + +- **They may not define any local types** (other than typealiases). +- **They must not reference any** `private` **entities,** except for + local functions declared within the inlineable function itself. +- **They must not reference any** `internal` **entities except for + those that have been** [availability-pinned](#pinning). See below + for a discussion of pinning. +- **They must not reference any entities less available than the + function itself.** + +An inlineable function is still emitted into its own module's binary. +This makes it possible to take an existing function and make it +inlineable, as long as the current body makes sense when deploying +against an earlier version of the library. + +If the body of an inlineable function is used in any way by a client +module (say, to determine that it does not read any global variables), +that module must take care to emit and use its own copy of the function. +This is because analysis of the function body may not apply to the +version of the function currently in the library. + +### Local Functions + +If an inlineable function contains local functions or closures, these +are implicitly made inlineable as well. This is important in case you +decide to change the inlineable function later. If the inlineable +function is emitted into a client module as described above, the local +functions must be as well. (At the SIL level, these local functions are +considered to have `shared` linkage.) + +### Pinning + +An [availability-pinned](#availability-pinned) entity is simply an `internal` member, free +function, or global binding that has been marked `@available`. This +promises that the entity will be available at link time in the +containing module's binary. This makes it safe to refer to such an +entity from an inlineable function. If a pinned entity is ever made +`public`, its availability should not be changed. + +> **note** +> +> Why isn't this a special form of `public`? Because we don't want it to +> imply everything that `public` does, such as requiring overrides to be +> `public`. + +Because a pinned class member may eventually be made public, it must be +assumed that new overrides may eventually appear from outside the module +unless the member is marked `final` or the class is not publicly +subclassable. + +We could do away with the entire "pinning" feature if we restricted +inlineable functions to only refer to public entities. However, this +removes one of the primary reasons to make something inlineable: to +allow efficient access to a type while still protecting its invariants. + +> **note** +> +> Types are not allowed to be pinned because that would have many more +> ripple effects. It's not technically impossible; it just requires a +> lot more thought. + +A Unifying Theme +---------------- + +So far this proposal has talked about three separate ways to lock down +on three separate Swift entities: structs, enums, and functions. Each of +these has a different set of constraints it enforces on the library +author and promises it makes to clients. However, they all follow a +common theme of giving up the flexibility of future changes in exchange +for improved performance and perhaps some semantic guarantees. As such, +we could consider using a common attribute, say `@fixed`, `@inline`, or +`@fragile`; either way, all attributes in this section can be referred +to as "fragility attributes". + +Constants +--------- + +The `let` keyword creates a named constant whose value will not change +for the lifetime of that entity; for a global or static constant, this +lasts from when the constant is first accessed (and lazily initialized) +until the end of program execution. However, different versions of the +same library may choose to have different values for a constant---say, a +string describing the library's copyright information. + +In order to make use of a constant's value across library boundaries, +the library owner may mark the constant as `@inlineable`. As when +applied to functions, the attribute takes a version number specifying +which versions of the library will behave correctly if the value is +inlined into client code. + +Note that if the constant's initial value expression has any observable +side effects, including the allocation of class instances, it must not +be treated as inlineable. A constant must always behave as if it is +initialized exactly once. + +> **TODO** +> +> Is this a condition we can detect at compile-time? Do we have to be +> restricted to things that can be lowered to compile-time constants? + +Properties +---------- + +By default, a stored property in a struct or class may be replaced by a +computed property in later versions of a library. As shown above, the +`@fixed_layout` attribute promises that all stored properties currently +in a type will remain stored in all future library versions, but +sometimes that isn't a reasonable promise. In this case, a library owner +may still want to allow clients to rely on a *specific* stored property +remaining stored, by applying the `@fixed` attribute to the property. + +> **TODO** +> +> Is it valid for a fixed property to have observing accessors, or is it +> more useful to promise that the setter is just a direct field access +> too? If it were spelled `@fragile`, I would assume that accessors are +> permitted but they become inlineable, and so not having any accessors +> is just a degenerate case of that. +> +> Is this feature sufficiently useful to be proposed initially at all, +> or is it too granular? + +Like all other attributes in this section, the `@fixed` attribute must +specify in which version of the library clients may rely on the property +being stored. The attribute may not be applied to non-final properties +in classes. + +> **note** +> +> It would be possible to allow `@fixed` on non-final properties, and +> have it only apply when the client code is definitively working with +> an instance of the base class, not any of its subclasses. But this is +> probably too subtle, and makes it look like the attribute is doing +> something useful when it actually isn't. + +Other Promises About Types +-------------------------- + +Advanced users may want to promise more specific things about various +types. These are similar to the internal `effects` attribute we have for +functions, except that they can be enforced by the compiler. + +- [`trivial`](#trivial): Promises that the type is [trivial](#trivial). Note + that this is not a recursive property; a trivial type may still require + indirection due to having an unknown size, and so a type containing that + type is not considered trivial. +- `size_in_bits(N)`: Promises that the type is not larger than a + certain size. (It may be smaller.) +- `no_payload`: Promises that an enum does not have payloads on any of + its cases (even the non-public ones). + +Collectively these features are known as "performance assertions", to +underscore the fact that they do not affect how a type is used at the +source level, but do allow for additional optimizations. We may also +expose some of these qualities to static or dynamic queries for +performance-sensitive code. + +All of these features take a version number, just like the more semantic +fragility attributes above. The exact spelling is not proposed by this +document. + +Optimization +============ + +Allowing a library to evolve inhibits the optimization of client code in +several ways. For example: + +- A function that currently does not access global memory might do so + in the future, so calls to it cannot be freely reordered in + client code. +- A stored property may be replaced by a computed property in the + future, so client code must not try to access the storage directly. +- A struct may have additional members in the future, so client code + must not assume it fits in any fixed-sized allocation. + +In order to make sure client code doesn't make unsafe assumptions, +queries about properties that may change between library versions must +be parameterized with the [availability context](#availability-context) that is +using the entity. An availability context is a set of minimum platform and +library versions that can be assumed present for code executing within the +context. This allows the compiler to answer the question, "Given what I +know about where this code will be executed, what can I assume about a +particular entity being used?". + +If the entity is declared within the same module as the code that's +using it, then the code is permitted to know all the details of how the +entity is declared. After all, if the entity is changed, the code that's +using it will be recompiled. + +However, if the entity is declared in another module, then the code +using it must be more conservative, and will therefore receive more +conservative answers to its queries. For example, a stored property may +report itself as computed. + +The presence of versioned fragility attributes makes the situation more +complicated. Within a client function that requires version 1.5 of a +particular library, the compiler should be able to take advantage of any +fragility information (and performance assertions) introduced prior to +version 1.5. + +Inlineable Code +--------------- + +By default, the availability context for a library always includes the +latest version of the library itself, since that code is always +distributed as a unit. However, this is not true for functions that have +been marked inlineable (see [Inlineable Functions](#inlineable-functions) +above). Inlineable code must be treated as if it is outside the current module, +since once it's inlined it will be. + +For inlineable code, the availability context is exactly the same as the +equivalent non-inlineable code except that the assumed version of the +containing library is the version attached to the `@inlineable` +attribute. Code within this context must be treated as if the containing +library were just a normal dependency. + +A publicly inlineable function still has a public symbol, which may be +used when the function is referenced from a client rather than called. +This version of the function is not subject to the same restrictions as +the version that may be inlined, and so it may be desirable to compile a +function twice: once for inlining, once for maximum performance. + +Local Availability Contexts +--------------------------- + +Swift availability contexts aren't just at the declaration level; they +also cover specific regions of code inside function bodies as well. +These "local" constructs are formed using the `#available` construct, +which performs a dynamic check. + +In theory, it would be legal to allow code dominated by a `#available` +check to take advantage of additional fragility information introduced +by the more restrictive dependencies that were checked for. However, +this is an additional optimization that may be complicated to implement +(and even to represent properly in SIL), and so it is not a first +priority. + +Resilience Domains +================== + +As described in the [Introduction](#introduction), the features and considerations +discussed in this document do not apply to libraries distributed in a +bundle with their clients. In this case, a client can rely on all the +current implementation details of its libraries when compiling, since +the same version of the library is guaranteed to be present at runtime. +This allows more optimization than would otherwise be possible. + +In some cases, a collection of libraries may be built and delivered +together, even though their clients may be packaged separately. (For +example, the ICU project is usually built into several library binaries, +but these libraries are always distributed together.) While the +*clients* cannot rely on a particular version of any library being +present, the various libraries in the collection should be able to take +advantage of the implementations of their dependencies also in the +collection---that is, it should treat all entities as if marked with the +appropriate fragility attributes. Modules in this sort of collection are +said to be in the same *resilience domain.* + +Exactly how resilience domains are specified is not covered by this +proposal, and indeed they are an additive feature. One possibility is +that a library's resilience domain defaults to the name of the module, +but can be overridden. If a client has the same resilience domain name +as a library it is using, it may assume that version of the library will +be present at runtime. + +Protocol Conformances +===================== + +Consider this scenario: a library is released containing both a +`MagicType` protocol and a `Wand` struct. `Wand` satisfies all the +requirements of the `MagicType` protocol, but the conformance was never +actually declared in the library. Someone files a bug, and it gets fixed +in version 1.1. + +Now, what happens when this client code is deployed against version 1.0 +of the library? + + // Library + @available(1.0) + public func classifyMagicItem(item: Item) -> MagicKind + + // Client + let kind = classifyMagicItem(elderWand) + log("\(elderWand): \(kind)") + +In order to call `classifyMagicItem`, the client code needs access to +the conformance of `Wand` to the `MagicType` protocol. But that +conformance *didn't exist* in version 1.0, so the client program will +fail on older systems. + +Therefore, a library author needs a way to declare that a type *now* +conforms to a protocol when it previously didn't. The way to do this is +by placing availability information on an extension: + + @available(1.1) + extension Wand : MagicType {} + +Note that this is unnecessary if either `Wand` or `MagicType` were +itself introduced in version 1.1; in that case, it would not be possible +to access the conformance from a context that only required 1.0. + +As with access control, applying `@available` to an extension overrides +the default availability of entities declared within the extension; +unlike access control, entities within the extension may freely declare +themselves to be either more or less available than what the extension +provides. + +> **note** +> +> This may feel like a regression from Objective-C, where [duck +> typing](#duck-typing) would allow a `Wand` to be passed as an `id ` +> without ill effects. However, `Wand` would still fail a `-conformsToProtocol:` +> check in version 1.0 of the library, and so whether or not the client +> code will work is dependent on what should be implementation details +> of the library. + +Checking Binary Compatibility +============================= + +With this many manual controls, it's important that library owners be +able to check their work. Therefore, we intend to ship a tool that can +compare two versions of a library's public interface, and present any +suspect differences for verification. Important cases include but are +not limited to: + +- Removal of public entities. +- Incompatible modifications to public entities, such as added + protocol conformances lacking versioning information. +- Unsafely-backdated "fragile" attributes as discussed in the + [Giving Up Flexibility](#giving-up-flexibility) section. +- Unsafe modifications to entities marked with the "fragile" + attributes, such as adding a stored property to a + `@fixed_layout` struct. + +Automatic Versioning +-------------------- + +A possible extension of this "checker" would be a tool that +*automatically* generates versioning information for entities in a +library, given the previous public interface of the library. This would +remove the need for versions on any of the fragility attributes, and +declaring versioned API would be as simple as marking an entity +`public`. Obviously this would also remove the possibility of human +error in managing library versions. + +However, making this tool has a number of additional difficulties beyond +the simple checker tool: + +- The tool must be able to read past library interface formats. This + is true for a validation tool as well, but the cost of failure is + much higher. Similarly, the past version of a library *must* be + available to correctly compile a new version. +- Because the information goes into a library's public interface, the + versioning tool must either be part of the compilation process, + modify the interface generated by compilation, or produce a sidecar + file that can be loaded when compiling the client. In any case, it + must *produce* information in addition to *consuming* it. +- Occasionally a library owner may want to override the + inferred versions. This can be accomplished by providing explicit + versioning information, as in the proposal. +- Bugs in the tool manifest as bugs in client programs. + +Because this tool would require a fair amount of additional work, it is +not part of this initial model. It is something we may decide to add in +the future. + +Summary +======= + +When possible, Swift gives library developers freedom to evolve their +code without breaking binary compatibility. This has implications for +both the semantics and performance of client code, and so library owners +also have tools to waive the ability to make certain future changes. The +language guarantees that client code will never accidentally introduce +implicit dependencies on specific versions of libraries. + +Glossary +======== + +##### ABI + - The runtime contract for using a particular API (or for an entire library), + including things like symbol names, calling conventions, and type layout + information. Stands for "Application Binary Interface". + +##### API + - An [entity](#entity) in a library that a [client](#client) may use, or the collection of all + such entities in a library. (If contrasting with [SPI](#spi), only those entities + that are available to arbitrary clients.) Marked `public` in + Swift. Stands for "Application Programming Interface". + +##### availability context + - The collection of library and platform versions that can be assumed, at + minimum, to be present in a certain block of code. Availability contexts + are always properly nested, and the global availability context includes + the module's minimum deployment target and minimum dependency versions. + +##### availability-pinned + - See [Pinning](#pinning). + +##### backwards-compatible + - A modification to an API that does not break existing clients. May also + describe the API in question. + +##### binary compatibility + - A general term encompassing both backwards- and forwards-compatibility + concerns. Also known as "ABI compatibility". + +##### client + - A target that depends on a particular library. It's usually easiest to + think of this as an application, but it could be another library. + (In certain cases, the "library" is itself an application, such as when + using Xcode's unit testing support.) + +##### duck typing + - In Objective-C, the ability to treat a class instance as having an + unrelated type, as long as the instance handles all messages sent to it. + (Note that this is a dynamic constraint.) + +##### entity + - A type, function, member, or global in a Swift program. + +##### forwards-compatible + - An API that is designed to handle future clients, perhaps allowing certain + changes to be made without changing the ABI. + +##### fragility attribute + - See [A Unifying Theme](#a-unifying-theme). + +##### module + - The primary unit of code sharing in Swift. Code in a module is always built + together, though it may be spread across several source files. + +##### performance assertion + - See [Other Promises About Types](#other-promises-about-types). + +##### resilience domain + - A grouping for code that will always be recompiled and distributed + together, and can thus take advantage of details about a type + even if it changes in the future. + +##### SPI + - A subset of [API](#api) that is only available to certain clients. Stands for + "System Programming Interface". + +##### target + - In this document, a collection of code in a single Swift module that is + built together; a "compilation unit". Roughly equivalent to a target in + Xcode. + +##### trivial + - A value whose assignment just requires a fixed-size bit-for-bit copy + without any indirection or reference-counting operations. diff --git a/docs/LibraryEvolution.rst b/docs/LibraryEvolution.rst deleted file mode 100644 index d20044a9077c4..0000000000000 --- a/docs/LibraryEvolution.rst +++ /dev/null @@ -1,776 +0,0 @@ -:orphan: - -.. default-role:: term -.. title:: Library Evolution Support in Swift ("Resilience") - -:Author: Jordan Rose -:Author: John McCall - -One of Swift's primary design goals is to allow efficient execution of code -without sacrificing load-time abstraction of implementation. - -Abstraction of implementation means that code correctly written against a -published interface will correctly function when the underlying implementation -changes to anything which still satisfies the original interface. There are -many potential reasons to provide this sort of abstraction. Apple's primary -interest is in making it easy and painless for our internal and external -developers to improve the ecosystem of Apple products by creating good and -secure programs and libraries; subtle deployment problems and/or unnecessary -dependencies on the behavior of our implementations would work against these -goals. - -Our current design in Swift is to provide opt-out load-time abstraction of -implementation for all language features. Alone, this would either incur -unacceptable cost or force widespread opting-out of abstraction. We intend to -mitigate this primarily by designing the language and its implementation to -minimize unnecessary and unintended abstraction: - -* Within the domain that defines an entity, all the details of its - implementation are available. - -* When entities are not exposed outside their defining module, their - implementation is not constrained. - -* By default, entities are not exposed outside their defining modules. This is - independently desirable to reduce accidental API surface area, but happens to - also interact well with the performance design. - -* Avoiding unnecessary language guarantees and taking advantage of that - flexibility to limit load-time costs. - -We also intend to provide tools to detect inadvertent changes in interfaces. - -.. contents:: :local: - -.. warning:: **This document is still in draft stages.** Large additions and - restructuring are still planned, including: - - * A summary for each declaration kind what changes are binary-compatible. - * A proper definition for "versioned entity". - * Several possible versioned attribute syntaxes, instead of just this one. - * A discussion of back-dating, and how it usually is not allowed. - * A brief discussion of the implementation issues for fixed-layout value types with resilient members, and with non-public members. - * A revisal of the discussion on fixed-layout classes. - * A brief discussion of "deployment files", which represent distribution groupings that are themselves versioned. (For example, OS X 10.10.3 contains Foundation version 1153.20.) Deployment files are likely to provide a concrete implementation of "resilience domains". - * A way to specify "minimum deployment libraries", like today's minimum deployment targets. - -Introduction -============ - -This model is intended to serve library designers whose libraries will evolve -over time. Such libraries must be both `backwards-compatible`, meaning that -existing clients should continue to work even when the library is updated, and -`forwards-compatible`, meaning that future clients will be able run using the -current version of the library. In simple terms: - -- Last year's apps should work with this year's library. -- Next year's apps should work with this year's library. - -This document will frequently refer to a *library* which vends public APIs, and -a single *client* that uses them. The same principles apply even when multiple -libraries and multiple clients are involved. - -This model is not of interest to libraries that are bundled with their clients -(distribution via source, static library, or embedded/sandboxed dynamic -library). Because a client always uses a particular version of such a library, -there is no need to worry about backwards- or forwards-compatibility. Just as -developers with a single app target are not forced to think about access -control, anyone writing a bundled library should not be required to use any of -the annotations described below in order to achieve full performance. - -The term "resilience" comes from the occasional use of "fragile" to describe -certain constructs that have very strict binary compatibility rules. For -example, a client's use of a C struct is "fragile" in that if the library -changes the fields in the struct, the client's use will "break". In Swift, -changing the fields in a struct will not automatically cause problems for -existing clients, so we say the struct is "resilient". - - -Using Versioned API -=================== - -References to a versioned API must always be guarded with the appropriate -availability checks. This means that any client entities that rely on having -certain APIs from a library must themselves be restricted to contexts in which -those APIs are available. This is accomplished using ``@available`` as well, -by specifying the name of the client library along with the required version:: - - // Client code - @available(Magician 1.5) - class CrystalBallView : MagicView { … } - -Library versions can also be checked dynamically using ``#available``, allowing -for fallback behavior when the requested library version is not present:: - - func scareMySiblings() { - if #available(Magician 1.2) { - conjureDemons() - } else { - print("BOO!!") - } - } - -.. note:: - - Possible implementations include generating a hidden symbol into a library, - or putting the version number in some kind of metadata, like the Info.plist - in a framework bundle on Darwin platforms. - -This is essentially the same model as the availability checking released in -Swift 2.0, but generalized for checking library versions instead of just OS -versions. - - -Publishing Versioned API -======================== - -A library's API is already marked with the ``public`` attribute. Versioning -information can be added to any ``public`` entity with the ``@available`` -attribute, this time specifying *only* a version number. This declares when the -entity was first exposed publicly in the current module. - -:: - - @available(1.2) - public func conjureDemons() - -.. admonition:: TODO - - Should this go on ``public`` instead? How does this play with SPI - ? - -Using the same attribute for both publishing and using versioned APIs helps tie -the feature together and enforces a consistent set of rules. The one difference -is that code within a library may always use all other entities declared within -the library (barring their own availability checks), since the entire library -is shipped as a unit. That is, even if a particular API was introduced in v1.0, -its (non-public) implementation may refer to APIs introduced in later versions. - -Swift libraries are strongly encouraged to use `semantic versioning`_, but this -is not enforced by the language. - -Some ``internal`` entities may also use ``@available``. See `Pinning`_ below. - -.. _semantic versioning: http://semver.org - - -Giving Up Flexibility -===================== - -Fixed-layout Structs -~~~~~~~~~~~~~~~~~~~~ - -By default, a library owner may add members to a public struct between releases -without breaking binary compatibility. This requires a certain amount of care -and indirection when dealing with values of struct type to account for the -struct's size and non-`trivial` fields not being known in advance, which of -course has performance implications. - -To opt out of this flexibility, a struct may be marked ``@fixed_layout``. This -promises that no stored properties will be added to or removed from the struct, -even ``private`` or ``internal`` ones. Methods and computed properties may -still be added to the struct. - -The ``@fixed_layout`` attribute takes a version number, just like -``@available``. This is so that clients can deploy against older versions of -the library, which may have a different layout for the struct. (In this case -the client must manipulate the struct as if the ``@fixed_layout`` attribute -were absent.) - -.. admonition:: TODO - - There's a benefit to knowing that a struct was ``@fixed_layout`` since it - was first made available. How should that be spelled? - - -Fixed-layout Classes? ---------------------- - -There is some benefit to knowing that a class has a fixed layout---that is, -that the stored properties of the class and all its superclasses are guaranteed -not to change in future versions of a library. This would, for example, allow -the class's memory to be allocated on the stack, as long as it can be proven -that no references to the class escape. However, such a constraint is unlikely -to be provable in practice from outside the class's own module, where its -primary operations are opaquely defined. Thus, until a tangible benefit has -been demonstrated, the ``@fixed_layout`` attribute will not apply to classes. - -(Another benefit would be to simplify the calculations needed for the offsets -of stored properties within classes. However, it's unclear that this would have -any significant benefit, particularly when most public properties are -manipulated through their accessors.) - - -Closed Enums -~~~~~~~~~~~~ - -By default, a library owner may add new cases to a public enum between releases -without breaking binary compatibility. As with structs, this results in a fair -amount of indirection when dealing with enum values, in order to potentially -accommodate new values. - -.. note:: - - If an enum value has a known case, or can be proven to belong to a set of - known cases, the compiler is of course free to use a more efficient - representation for the value, just as it may discard fields of structs that - are provably never accessed. - -A library owner may opt out of this flexibility by marking the enum as -``@closed``. A "closed" enum may not have any ``private`` or ``internal`` cases -and may not add new cases in the future. This guarantees to clients that the -enum cases are exhaustive. - -.. note:: - - Were a "closed" enum allowed to have non-public cases, clients of the - library would still have to treat the enum as opaque and would still have - to be able to handle unknown cases in their ``switch`` statements. - -The ``@closed`` attribute takes a version number, just like ``@available``. -This is so that clients can deploy against older versions of the library, which -may have non-public cases in the enum. (In this case the client must manipulate -the enum as if the ``@closed`` attribute were absent.) - -Even for default "open" enums, adding new cases should not be done lightly. Any -clients attempting to do an exhaustive switch over all enum cases will likely -not handle new cases well. - -.. note:: - - One possibility would be a way to map new cases to older ones on older - clients. This would only be useful for certain kinds of enums, though, and - adds a lot of additional complexity, all of which would be tied up in - versions. Our generalized switch patterns probably make it hard to nail - down the behavior here. - - -Inlineable Functions -~~~~~~~~~~~~~~~~~~~~ - -Functions are a very common example of resilience: the function's declaration -is published as API, but its body may change between library versions as long -as it upholds the same semantic contracts. This applies to other function-like -constructs as well: initializers, accessors, and deinitializers. - -However, sometimes it is useful to provide the body to clients as well. There -are a few common reasons for this: - -- The function only performs simple operations, and so inlining it will both - save the overhead of a cross-library function call and allow further - optimization of callers. - -- The function accesses a fixed-layout struct with non-public members; this - allows the library author to preserve invariants while still allowing - efficient access to the struct. - -A public function marked with the ``@inlineable`` attribute makes its body -available to clients as part of the module's public interface. The -``@inlineable`` attribute takes a version number, just like ``@available``; -clients may not assume that the body of the function is suitable when deploying -against older versions of the library. - -Clients are not required to inline a function marked ``@inlineable``. - -.. note:: - - It is legal to change the implementation of an inlineable function in the - next release of the library. However, any such change must be made with the - understanding that it may or may not affect existing clients. - -Restrictions ------------- - -Because the body of an inlineable function (or method, accessor, initializer, -or deinitializer) may be inlined into another module, it must not make any -assumptions that rely on knowledge of the current module. Here is a trivial -example:: - - public struct Point2D { - var x, y: Double - public init(x: Double, y: Double) { … } - } - - extension Point2D { - @inlineable public func distanceTo(other: Point2D) -> Double { - let deltaX = self.x - other.x - let deltaY = self.y - other.y - return sqrt(deltaX*deltaX + deltaY*deltaY) - } - } - -As written, this ``distanceTo`` method is not safe to inline. The next release -of the library could very well replace the implementation of ``Point2D`` with a -polar representation:: - - public struct Point2D { - var r, theta: Double - public init(x: Double, y: Double) { … } - } - -and the ``x`` and ``y`` properties have now disappeared. To avoid this, we have -the following restrictions on the bodies of inlineable functions: - -- **They may not define any local types** (other than typealiases). - -- **They must not reference any** ``private`` **entities,** except for local - functions declared within the inlineable function itself. - -- **They must not reference any** ``internal`` **entities except for those that - have been** `availability-pinned`_. See below for a discussion of pinning. - -- **They must not reference any entities less available than the function - itself.** - -.. _availability-pinned: #pinning - -An inlineable function is still emitted into its own module's binary. This -makes it possible to take an existing function and make it inlineable, as long -as the current body makes sense when deploying against an earlier version of -the library. - -If the body of an inlineable function is used in any way by a client module -(say, to determine that it does not read any global variables), that module -must take care to emit and use its own copy of the function. This is because -analysis of the function body may not apply to the version of the function -currently in the library. - -Local Functions ---------------- - -If an inlineable function contains local functions or closures, these are -implicitly made inlineable as well. This is important in case you decide to -change the inlineable function later. If the inlineable function is emitted -into a client module as described above, the local functions must be as well. -(At the SIL level, these local functions are considered to have ``shared`` -linkage.) - -Pinning -------- - -An `availability-pinned` entity is simply an ``internal`` member, free -function, or global binding that has been marked ``@available``. This promises -that the entity will be available at link time in the containing module's -binary. This makes it safe to refer to such an entity from an inlineable -function. If a pinned entity is ever made ``public``, its availability should -not be changed. - -.. note:: - - Why isn't this a special form of ``public``? Because we don't want it to - imply everything that ``public`` does, such as requiring overrides to be - ``public``. - -Because a pinned class member may eventually be made public, it must be assumed -that new overrides may eventually appear from outside the module unless the -member is marked ``final`` or the class is not publicly subclassable. - -We could do away with the entire "pinning" feature if we restricted inlineable -functions to only refer to public entities. However, this removes one of the -primary reasons to make something inlineable: to allow efficient access to a -type while still protecting its invariants. - -.. note:: - - Types are not allowed to be pinned because that would have many more ripple - effects. It's not technically impossible; it just requires a lot more - thought. - - -A Unifying Theme -~~~~~~~~~~~~~~~~ - -So far this proposal has talked about three separate ways to lock down on three -separate Swift entities: structs, enums, and functions. Each of these has a -different set of constraints it enforces on the library author and promises it -makes to clients. However, they all follow a common theme of giving up the -flexibility of future changes in exchange for improved performance and perhaps -some semantic guarantees. As such, we could consider using a common attribute, -say ``@fixed``, ``@inline``, or ``@fragile``; either way, all attributes in -this section can be referred to as "fragility attributes". - - -Constants -~~~~~~~~~ - -The ``let`` keyword creates a named constant whose value will not change for -the lifetime of that entity; for a global or static constant, this lasts from -when the constant is first accessed (and lazily initialized) until the end of -program execution. However, different versions of the same library may choose -to have different values for a constant---say, a string describing the -library's copyright information. - -In order to make use of a constant's value across library boundaries, the -library owner may mark the constant as ``@inlineable``. As when applied to -functions, the attribute takes a version number specifying which versions of -the library will behave correctly if the value is inlined into client code. - -Note that if the constant's initial value expression has any observable side -effects, including the allocation of class instances, it must not be treated -as inlineable. A constant must always behave as if it is initialized exactly -once. - -.. admonition:: TODO - - Is this a condition we can detect at compile-time? Do we have to be - restricted to things that can be lowered to compile-time constants? - - -Properties -~~~~~~~~~~ - -By default, a stored property in a struct or class may be replaced by a -computed property in later versions of a library. As shown above, the -``@fixed_layout`` attribute promises that all stored properties currently in a -type will remain stored in all future library versions, but sometimes that -isn't a reasonable promise. In this case, a library owner may still want to -allow clients to rely on a *specific* stored property remaining stored, by -applying the ``@fixed`` attribute to the property. - -.. admonition:: TODO - - Is it valid for a fixed property to have observing accessors, or is it more - useful to promise that the setter is just a direct field access too? If it - were spelled ``@fragile``, I would assume that accessors are permitted but - they become inlineable, and so not having any accessors is just a - degenerate case of that. - - Is this feature sufficiently useful to be proposed initially at all, or is - it too granular? - -Like all other attributes in this section, the ``@fixed`` attribute must -specify in which version of the library clients may rely on the property being -stored. The attribute may not be applied to non-final properties in classes. - -.. note:: - - It would be possible to allow ``@fixed`` on non-final properties, and have - it only apply when the client code is definitively working with an instance - of the base class, not any of its subclasses. But this is probably too - subtle, and makes it look like the attribute is doing something useful when - it actually isn't. - - - -Other Promises About Types -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Advanced users may want to promise more specific things about various types. -These are similar to the internal ``effects`` attribute we have for functions, -except that they can be enforced by the compiler. - -- ``trivial``: Promises that the type is `trivial`. Note that this is not a - recursive property; a trivial type may still require indirection due to - having an unknown size, and so a type containing that type is not considered - trivial. - -- ``size_in_bits(N)``: Promises that the type is not larger than a certain - size. (It may be smaller.) - -- ``no_payload``: Promises that an enum does not have payloads on any of its - cases (even the non-public ones). - -Collectively these features are known as "performance assertions", to -underscore the fact that they do not affect how a type is used at the source -level, but do allow for additional optimizations. We may also expose some of -these qualities to static or dynamic queries for performance-sensitive code. - -All of these features take a version number, just like the more semantic -fragility attributes above. The exact spelling is not proposed by this document. - - -Optimization -============ - -Allowing a library to evolve inhibits the optimization of client code in -several ways. For example: - -- A function that currently does not access global memory might do so in the - future, so calls to it cannot be freely reordered in client code. - -- A stored property may be replaced by a computed property in the future, so - client code must not try to access the storage directly. - -- A struct may have additional members in the future, so client code must not - assume it fits in any fixed-sized allocation. - -In order to make sure client code doesn't make unsafe assumptions, queries -about properties that may change between library versions must be parameterized -with the `availability context` that is using the entity. An availability -context is a set of minimum platform and library versions that can be assumed -present for code executing within the context. This allows the compiler to -answer the question, "Given what I know about where this code will be executed, -what can I assume about a particular entity being used?". - -If the entity is declared within the same module as the code that's using it, -then the code is permitted to know all the details of how the entity is -declared. After all, if the entity is changed, the code that's using it will be -recompiled. - -However, if the entity is declared in another module, then the code using it -must be more conservative, and will therefore receive more conservative answers -to its queries. For example, a stored property may report itself as computed. - -The presence of versioned fragility attributes makes the situation more -complicated. Within a client function that requires version 1.5 of a particular -library, the compiler should be able to take advantage of any fragility -information (and performance assertions) introduced prior to version 1.5. - - -Inlineable Code -~~~~~~~~~~~~~~~ - -By default, the availability context for a library always includes the latest -version of the library itself, since that code is always distributed as a unit. -However, this is not true for functions that have been marked inlineable (see -`Inlineable Functions`_ above). Inlineable code must be treated as if it is -outside the current module, since once it's inlined it will be. - -For inlineable code, the availability context is exactly the same as the -equivalent non-inlineable code except that the assumed version of the -containing library is the version attached to the ``@inlineable`` attribute. -Code within this context must be treated as if the containing library were just -a normal dependency. - -A publicly inlineable function still has a public symbol, which may be used -when the function is referenced from a client rather than called. This version -of the function is not subject to the same restrictions as the version that -may be inlined, and so it may be desirable to compile a function twice: once -for inlining, once for maximum performance. - - -Local Availability Contexts -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Swift availability contexts aren't just at the declaration level; they also -cover specific regions of code inside function bodies as well. These "local" -constructs are formed using the ``#available`` construct, which performs a -dynamic check. - -In theory, it would be legal to allow code dominated by a ``#available`` check -to take advantage of additional fragility information introduced by the more -restrictive dependencies that were checked for. However, this is an additional -optimization that may be complicated to implement (and even to represent -properly in SIL), and so it is not a first priority. - - -Resilience Domains -================== - -As described in the `Introduction`_, the features and considerations discussed -in this document do not apply to libraries distributed in a bundle with their -clients. In this case, a client can rely on all the current implementation -details of its libraries when compiling, since the same version of the library -is guaranteed to be present at runtime. This allows more optimization than -would otherwise be possible. - -In some cases, a collection of libraries may be built and delivered together, -even though their clients may be packaged separately. (For example, the ICU -project is usually built into several library binaries, but these libraries are -always distributed together.) While the *clients* cannot rely on a particular -version of any library being present, the various libraries in the collection -should be able to take advantage of the implementations of their dependencies -also in the collection---that is, it should treat all entities as if marked -with the appropriate fragility attributes. Modules in this sort of collection -are said to be in the same *resilience domain.* - -Exactly how resilience domains are specified is not covered by this proposal, -and indeed they are an additive feature. One possibility is that a library's -resilience domain defaults to the name of the module, but can be overridden. If -a client has the same resilience domain name as a library it is using, it may -assume that version of the library will be present at runtime. - - -Protocol Conformances -===================== - -Consider this scenario: a library is released containing both a ``MagicType`` -protocol and a ``Wand`` struct. ``Wand`` satisfies all the requirements of the -``MagicType`` protocol, but the conformance was never actually declared in the -library. Someone files a bug, and it gets fixed in version 1.1. - -Now, what happens when this client code is deployed against version 1.0 of the -library? - -:: - - // Library - @available(1.0) - public func classifyMagicItem(item: Item) -> MagicKind - - // Client - let kind = classifyMagicItem(elderWand) - log("\(elderWand): \(kind)") - -In order to call ``classifyMagicItem``, the client code needs access to the -conformance of ``Wand`` to the ``MagicType`` protocol. But that conformance -*didn't exist* in version 1.0, so the client program will fail on older systems. - -Therefore, a library author needs a way to declare that a type *now* conforms -to a protocol when it previously didn't. The way to do this is by placing -availability information on an extension:: - - @available(1.1) - extension Wand : MagicType {} - -Note that this is unnecessary if either ``Wand`` or ``MagicType`` were itself -introduced in version 1.1; in that case, it would not be possible to access -the conformance from a context that only required 1.0. - -As with access control, applying ``@available`` to an extension overrides the -default availability of entities declared within the extension; unlike access -control, entities within the extension may freely declare themselves to be -either more or less available than what the extension provides. - -.. note:: - - This may feel like a regression from Objective-C, where `duck typing` would - allow a ``Wand`` to be passed as an ``id `` without ill effects. - However, ``Wand`` would still fail a ``-conformsToProtocol:`` check in - version 1.0 of the library, and so whether or not the client code will work - is dependent on what should be implementation details of the library. - - -Checking Binary Compatibility -============================= - -With this many manual controls, it's important that library owners be able to -check their work. Therefore, we intend to ship a tool that can compare two -versions of a library's public interface, and present any suspect differences -for verification. Important cases include but are not limited to: - -- Removal of public entities. - -- Incompatible modifications to public entities, such as added protocol - conformances lacking versioning information. - -- Unsafely-backdated "fragile" attributes as discussed in the `Giving Up - Flexibility`_ section. - -- Unsafe modifications to entities marked with the "fragile" attributes, such as - adding a stored property to a ``@fixed_layout`` struct. - - -Automatic Versioning -~~~~~~~~~~~~~~~~~~~~ - -A possible extension of this "checker" would be a tool that *automatically* -generates versioning information for entities in a library, given the previous -public interface of the library. This would remove the need for versions on any -of the fragility attributes, and declaring versioned API would be as simple as -marking an entity ``public``. Obviously this would also remove the possibility -of human error in managing library versions. - -However, making this tool has a number of additional difficulties beyond the -simple checker tool: - -- The tool must be able to read past library interface formats. This is true - for a validation tool as well, but the cost of failure is much higher. - Similarly, the past version of a library *must* be available to correctly - compile a new version. - -- Because the information goes into a library's public interface, the - versioning tool must either be part of the compilation process, modify the - interface generated by compilation, or produce a sidecar file that can be - loaded when compiling the client. In any case, it must *produce* information - in addition to *consuming* it. - -- Occasionally a library owner may want to override the inferred versions. This - can be accomplished by providing explicit versioning information, as in the - proposal. - -- Bugs in the tool manifest as bugs in client programs. - -Because this tool would require a fair amount of additional work, it is not -part of this initial model. It is something we may decide to add in the future. - - -Summary -======= - -When possible, Swift gives library developers freedom to evolve their code -without breaking binary compatibility. This has implications for both the -semantics and performance of client code, and so library owners also have tools -to waive the ability to make certain future changes. The language guarantees -that client code will never accidentally introduce implicit dependencies on -specific versions of libraries. - - -Glossary -======== - -.. glossary:: - - ABI - The runtime contract for using a particular API (or for an entire library), - including things like symbol names, calling conventions, and type layout - information. Stands for "Application Binary Interface". - - API - An `entity` in a library that a `client` may use, or the collection of all - such entities in a library. (If contrasting with `SPI`, only those entities - that are available to arbitrary clients.) Marked ``public`` in - Swift. Stands for "Application Programming Interface". - - availability context - The collection of library and platform versions that can be assumed, at - minimum, to be present in a certain block of code. Availability contexts - are always properly nested, and the global availability context includes - the module's minimum deployment target and minimum dependency versions. - - availability-pinned - See `Pinning`_. - - backwards-compatible - A modification to an API that does not break existing clients. May also - describe the API in question. - - binary compatibility - A general term encompassing both backwards- and forwards-compatibility - concerns. Also known as "ABI compatibility". - - client - A target that depends on a particular library. It's usually easiest to - think of this as an application, but it could be another library. - (In certain cases, the "library" is itself an application, such as when - using Xcode's unit testing support.) - - duck typing - In Objective-C, the ability to treat a class instance as having an - unrelated type, as long as the instance handles all messages sent to it. - (Note that this is a dynamic constraint.) - - entity - A type, function, member, or global in a Swift program. - - forwards-compatible - An API that is designed to handle future clients, perhaps allowing certain - changes to be made without changing the ABI. - - fragility attribute - See `A Unifying Theme`_. - - module - The primary unit of code sharing in Swift. Code in a module is always built - together, though it may be spread across several source files. - - performance assertion - See `Other Promises About Types`_. - - resilience domain - A grouping for code that will always be recompiled and distributed - together, and can thus take advantage of details about a type - even if it changes in the future. - - SPI - A subset of `API` that is only available to certain clients. Stands for - "System Programming Interface". - - target - In this document, a collection of code in a single Swift module that is - built together; a "compilation unit". Roughly equivalent to a target in - Xcode. - - trivial - A value whose assignment just requires a fixed-size bit-for-bit copy - without any indirection or reference-counting operations. diff --git a/docs/Literals.md b/docs/Literals.md new file mode 100644 index 0000000000000..a2755b3bf60ea --- /dev/null +++ b/docs/Literals.md @@ -0,0 +1,131 @@ +Literals +======== + +*What happens when a literal expression is used?* + +The complicated case is for integer, floating-point, character, and +string literals, so let's look at those. + +High-Level View +--------------- + + window.setTitle("Welcome to Xcode") + +In this case, we have a string literal and an enclosing context. If +`window` is an NSWindow, there will only be one possible method named +`setTitle`, which takes an NSString. Therefore, we want the string +literal expression to end up being an NSString. + +Fortunately, NSString implements StringLiteralConvertible, so the type +checker will indeed be able to choose NSString as the type of the string +literal. All is well. + +In the case of integers or floating-point literals, the value +effectively has infinite precision. Once the type has been chosen, the +value is checked to see if it is in range for that type. + +The StringLiteralConvertible Protocol +------------------------------------- + +Here is the StringLiteralConvertible protocol as defined in the standard +library's Policy.swift: + + // NOTE: the compiler has builtin knowledge of this protocol + protocol StringLiteralConvertible { + typealias StringLiteralType : _BuiltinStringLiteralConvertible + class func convertFromStringLiteral(value : StringLiteralType) -> Self + } + +Curiously, the protocol is not defined in terms of primitive types, but +in terms of any StringLiteralType that the implementer chooses. In most +cases, this will be Swift's own native String type, which means users +can implement their own StringLiteralConvertible types while still +dealing with a high-level interface. + +(Why is this not hardcoded? A String *must* be a valid Unicode string, +but if the string literal contains escape sequences, an invalid series +of code points could be constructed...which may be what's desired in +some cases.) + +The \_BuiltinStringLiteralConvertible Protocol +---------------------------------------------- + +Policy.swift contains a second protocol: + + // NOTE: the compiler has builtin knowledge of this protocol + protocol _BuiltinStringLiteralConvertible { + class func _convertFromBuiltinStringLiteral(value : Builtin.RawPointer, + byteSize : Builtin.Int64, + isASCII: Builtin.Int1) -> Self + } + +The use of builtin types makes it clear that this is *only* for use in +the standard library. This is the actual primitive function that is used +to construct types from string literals: the compiler knows how to emit +raw data from the literal, and the arguments describe that raw data. + +So, the general runtime behavior is now clear: + +1. The compiler generates raw string data. +2. Some type conforming to \_BuiltinStringLiteralConvertible is + constructed from the raw string data. This will be a standard + library type. +3. Some type conforming to StringLiteralConvertible is constructed from + the object constructed in step 2. This may be a user-defined type. + This is the result. + +The Type-Checker's Algorithm +---------------------------- + +In order to make this actually happen, the type-checker has to do some +fancy footwork. Remember, at this point all we have is a string literal +and an expected type; if the function were overloaded, we would have to +try all the types. + +This algorithm can go forwards or backwards, since it's actually defined +in terms of constraints, but it's easiest to understand as a linear +process. + +1. Filter the types provided by the context to only include those that + are StringLiteralConvertible. +2. Using the associated StringLiteralType, find the appropriate + `_convertFromBuiltinStringLiteral`. +3. Using the type from step 1, find the appropriate + `convertFromStringLiteral`. +4. Build an expression tree with the appropriate calls. + +How about cases where there is no context? : + + var str = "abc" + +Here we have nothing to go on, so instead the type checker looks for a +global type named `StringLiteralType` in the current module-scope +context, and uses that type if it is actually a StringLiteralConvertible +type. This both allows different standard libraries to set different +default literal types, and allows a user to *override* the default type +in their own source file. + +The real story is even more complicated because of implicit conversions: +the type expected by `setTitle` might not actually be +literal-convertible, but something else that *is* literal-convertible +can then implicitly convert to the proper type. If this makes your head +spin, don't worry about it. + +Arrays, Dictionaries, and Interpolation +--------------------------------------- + +Array and dictionary literals don't have a Builtin\*Convertible form. +Instead, they just always use a variadic list of elements (`T...`) in +the array case and (key, value) tuples in the dictionary case. A +variadic list is always exposed using the standard library's Array type, +so there is no separate step to jump through. + +The default array literal type is always Array, and the default +dictionary literal type is always Dictionary. + +String interpolations are a bit different: they try to individually +convert each element of the interpolation to the type that adopts +StringInterpolationConvertible, then calls the variadic +`convertFromStringInterpolation` to put them all together. The default +type for an interpolated literal without context is also +`StringLiteralType`. diff --git a/docs/Literals.rst b/docs/Literals.rst deleted file mode 100644 index 57473e765d91f..0000000000000 --- a/docs/Literals.rst +++ /dev/null @@ -1,136 +0,0 @@ -:orphan: - -.. @raise litre.TestsAreMissing - -Literals -======== - -*What happens when a literal expression is used?* - -The complicated case is for integer, floating-point, character, and string -literals, so let's look at those. - - -High-Level View ---------------- - -:: - - window.setTitle("Welcome to Xcode") - -In this case, we have a string literal and an enclosing context. If ``window`` -is an NSWindow, there will only be one possible method named ``setTitle``, -which takes an NSString. Therefore, we want the string literal expression to -end up being an NSString. - -Fortunately, NSString implements StringLiteralConvertible, so the type checker -will indeed be able to choose NSString as the type of the string literal. All -is well. - -In the case of integers or floating-point literals, the value effectively has -infinite precision. Once the type has been chosen, the value is checked to see -if it is in range for that type. - - -The StringLiteralConvertible Protocol -------------------------------------- - -Here is the StringLiteralConvertible protocol as defined in the standard -library's Policy.swift:: - - // NOTE: the compiler has builtin knowledge of this protocol - protocol StringLiteralConvertible { - typealias StringLiteralType : _BuiltinStringLiteralConvertible - class func convertFromStringLiteral(value : StringLiteralType) -> Self - } - -Curiously, the protocol is not defined in terms of primitive types, but in -terms of any StringLiteralType that the implementer chooses. In most cases, -this will be Swift's own native String type, which means users can implement -their own StringLiteralConvertible types while still dealing with a high-level -interface. - -(Why is this not hardcoded? A String *must* be a valid Unicode string, but -if the string literal contains escape sequences, an invalid series of code -points could be constructed...which may be what's desired in some cases.) - - -The _BuiltinStringLiteralConvertible Protocol ---------------------------------------------- - -Policy.swift contains a second protocol:: - - // NOTE: the compiler has builtin knowledge of this protocol - protocol _BuiltinStringLiteralConvertible { - class func _convertFromBuiltinStringLiteral(value : Builtin.RawPointer, - byteSize : Builtin.Int64, - isASCII: Builtin.Int1) -> Self - } - -The use of builtin types makes it clear that this is *only* for use in the -standard library. This is the actual primitive function that is used to -construct types from string literals: the compiler knows how to emit raw -data from the literal, and the arguments describe that raw data. - -So, the general runtime behavior is now clear: - -1. The compiler generates raw string data. -2. Some type conforming to _BuiltinStringLiteralConvertible is constructed from - the raw string data. This will be a standard library type. -3. Some type conforming to StringLiteralConvertible is constructed from the - object constructed in step 2. This may be a user-defined type. This is the - result. - - -The Type-Checker's Algorithm ----------------------------- - -In order to make this actually happen, the type-checker has to do some fancy -footwork. Remember, at this point all we have is a string literal and an -expected type; if the function were overloaded, we would have to try all the -types. - -This algorithm can go forwards or backwards, since it's actually defined in -terms of constraints, but it's easiest to understand as a linear process. - -1. Filter the types provided by the context to only include those that are - StringLiteralConvertible. -2. Using the associated StringLiteralType, find the appropriate - ``_convertFromBuiltinStringLiteral``. -3. Using the type from step 1, find the appropriate - ``convertFromStringLiteral``. -4. Build an expression tree with the appropriate calls. - -How about cases where there is no context? :: - - var str = "abc" - -Here we have nothing to go on, so instead the type checker looks for a global -type named ``StringLiteralType`` in the current module-scope context, and uses -that type if it is actually a StringLiteralConvertible type. This both allows -different standard libraries to set different default literal types, and allows -a user to *override* the default type in their own source file. - -The real story is even more complicated because of implicit conversions: -the type expected by ``setTitle`` might not actually be literal-convertible, -but something else that *is* literal-convertible can then implicitly convert -to the proper type. If this makes your head spin, don't worry about it. - - -Arrays, Dictionaries, and Interpolation ---------------------------------------- - -Array and dictionary literals don't have a Builtin*Convertible form. Instead, -they just always use a variadic list of elements (``T...``) in the array case -and (key, value) tuples in the dictionary case. A variadic list is always -exposed using the standard library's Array type, so there is no separate step -to jump through. - -The default array literal type is always Array, and the default dictionary -literal type is always Dictionary. - -String interpolations are a bit different: they try to individually convert -each element of the interpolation to the type that adopts -StringInterpolationConvertible, then calls the variadic -``convertFromStringInterpolation`` to put them all together. The default type -for an interpolated literal without context is also ``StringLiteralType``. diff --git a/docs/LogicalObjects.md b/docs/LogicalObjects.md new file mode 100644 index 0000000000000..6095b684b183a --- /dev/null +++ b/docs/LogicalObjects.md @@ -0,0 +1,90 @@ +Logical Objects +=============== + +Universal across programming languages, we have this concept of a value, +which is just some amount of fixed data. A value might be the int 5, or +a pair of the bool true and the int -20, or an NSRect with the component +values (0, 0, 400, 600), or whatever. + +In imperative languages we have this concept of an object. It's an +unfortunately overloaded term; here I'm using it like the standards do, +which is to say that it's a thing that holds a value, but which can be +altered at any time to hold a different value. It's tempting to use the +word variable instead, and a variable is indeed an object, but +"variable" implies all this extra stuff, like being its own independent, +self-contained thing, whereas we want a word that also covers fields and +array elements and what-have-you. So let's just suck it up and go with +"object". + +You might also call these "r-value" and "l-value". These have their own +connotations that I don't want to get into. Stick with "value" and +"object". + +In C and C++, every object is physical. It's actually a place in memory +somewhere. It's not necessarily easily addressable (because of +bitfields), and its lifetime may be tightly constrained (because of +temporaries or deallocation), but it's always memory. + +In Objective-C, properties and subscripting add an idea of a logical +object. The only way you can manipulate it is by calling a function +(with unrestricted extra arguments) to either fetch the current value or +update it with a new value. The logical object doesn't promise to behave +like a physical object, either: for example, you can set it, then +immediately get it, and the result might not match the value you set. + +Swift has logical objects as well. We have them in a few new places +(global objects can be logical), and sometimes we treat objects that are +really physical as logical (because resilience prevents us from assuming +physicality), and we're considering making some restrictions on how +different a logical object can be from a physical object (although +set-and-get would still be opaque), but otherwise they're pretty much +just like they are in Objective-C. + +That said, they do interact with some other language features in +interesting ways. + +For example, methods on value types have a this parameter. Usually +parameters are values, but this is actually an object: if I call a +method on an object, and the method modifies the value of this, I expect +it to modify the object I called the method on. This is the high-level +perspective of what \[inout\] really means: that what we're really +passing as a parameter is an object, not a value. With one exception, +everything that follows applies to any sort of \[inout\] parameter, not +just this on value types. More on that exception later. + +How do you actually pass an object, though, given that even physical +objects might not be addressable, but especially given that an object +might be logical? + +Well, we can always treat a physical object like a logical object. It's +possible to come up with ways to implement passing a logical object +(pass a pointer to a struct, the first value of which is a getter, the +second value of which is a setter, and the rest of which is opaque to +the callee; the struct must be passed to the getter and setter +functions). Unfortunately, the performance implications would be +terrible: accessing multiple fields would involve multiple calls to the +getter, each returning tons of extra information. And getter and setter +calls might be very expensive. + +We could pass a hybrid format, using direct accesses when possible and a +getter/setter when not. Unfortunately, that's a lot of code bloat in +every single method implementation. + +Or we can always pass a physical, addressable object. This avoids +penalizing the fast case where the object is really physical, which is +great. For the case where the object is logical, we just have to make it +physical somehow. That means materialization: calling the getter, +storing the result into temporary memory, passing the temporary, and +then calling the setter with the new value in the temporary when the +method call is done. This last step is called writeback. + +(About that one exception to this all applying equally to \[inout\]: in +addition to all this stuff about calling methods on different kinds of +object, we also want to support calling a method on a value. This is +also implemented with a form of materialization, which looks just like +the logical-object kind except without writeback, because there's +nothing to write back to. This is a special rule that only applies to +passing this, because we assume that most types will have lots of useful +methods that don't involve writing to this, whereas we assume that a +function with an explicit \[inout\] parameter is almost certain to want +to write to it.) diff --git a/docs/LogicalObjects.rst b/docs/LogicalObjects.rst deleted file mode 100644 index a5fef5d4f85e0..0000000000000 --- a/docs/LogicalObjects.rst +++ /dev/null @@ -1,84 +0,0 @@ -.. @raise litre.TestsAreMissing - -Logical Objects -=============== - -Universal across programming languages, we have this concept of a value, which -is just some amount of fixed data. A value might be the int 5, or a pair of the -bool true and the int -20, or an NSRect with the component values (0, 0, 400, -600), or whatever. - -In imperative languages we have this concept of an object. It's an -unfortunately overloaded term; here I'm using it like the standards do, which is -to say that it's a thing that holds a value, but which can be altered at any -time to hold a different value. It's tempting to use the word variable instead, -and a variable is indeed an object, but "variable" implies all this extra stuff, -like being its own independent, self-contained thing, whereas we want a word -that also covers fields and array elements and what-have-you. So let's just -suck it up and go with "object". - -You might also call these "r-value" and "l-value". These have their own -connotations that I don't want to get into. Stick with "value" and "object". - -In C and C++, every object is physical. It's actually a place in memory -somewhere. It's not necessarily easily addressable (because of bitfields), and -its lifetime may be tightly constrained (because of temporaries or -deallocation), but it's always memory. - -In Objective-C, properties and subscripting add an idea of a logical object. -The only way you can manipulate it is by calling a function (with unrestricted -extra arguments) to either fetch the current value or update it with a new -value. The logical object doesn't promise to behave like a physical object, -either: for example, you can set it, then immediately get it, and the result -might not match the value you set. - -Swift has logical objects as well. We have them in a few new places (global -objects can be logical), and sometimes we treat objects that are really physical -as logical (because resilience prevents us from assuming physicality), and we're -considering making some restrictions on how different a logical object can be -from a physical object (although set-and-get would still be opaque), but -otherwise they're pretty much just like they are in Objective-C. - -That said, they do interact with some other language features in interesting -ways. - -For example, methods on value types have a this parameter. Usually parameters -are values, but this is actually an object: if I call a method on an object, and -the method modifies the value of this, I expect it to modify the object I called -the method on. This is the high-level perspective of what [inout] really means: -that what we're really passing as a parameter is an object, not a value. With -one exception, everything that follows applies to any sort of [inout] parameter, -not just this on value types. More on that exception later. - -How do you actually pass an object, though, given that even physical objects -might not be addressable, but especially given that an object might be logical? - -Well, we can always treat a physical object like a logical object. It's -possible to come up with ways to implement passing a logical object (pass a -pointer to a struct, the first value of which is a getter, the second value of -which is a setter, and the rest of which is opaque to the callee; the struct -must be passed to the getter and setter functions). Unfortunately, the -performance implications would be terrible: accessing multiple fields would -involve multiple calls to the getter, each returning tons of extra information. -And getter and setter calls might be very expensive. - -We could pass a hybrid format, using direct accesses when possible and a -getter/setter when not. Unfortunately, that's a lot of code bloat in every -single method implementation. - -Or we can always pass a physical, addressable object. This avoids penalizing -the fast case where the object is really physical, which is great. For the case -where the object is logical, we just have to make it physical somehow. That -means materialization: calling the getter, storing the result into temporary -memory, passing the temporary, and then calling the setter with the new value in -the temporary when the method call is done. This last step is called writeback. - -(About that one exception to this all applying equally to [inout]: in addition -to all this stuff about calling methods on different kinds of object, we also -want to support calling a method on a value. This is also implemented with a -form of materialization, which looks just like the logical-object kind except -without writeback, because there's nothing to write back to. This is a special -rule that only applies to passing this, because we assume that most types will -have lots of useful methods that don't involve writing to this, whereas we -assume that a function with an explicit [inout] parameter is almost certain to -want to write to it.) diff --git a/docs/Modules.md b/docs/Modules.md new file mode 100644 index 0000000000000..6a0062be1b6f3 --- /dev/null +++ b/docs/Modules.md @@ -0,0 +1,406 @@ +A module is the primary unit of code sharing in Swift. This document +describes the experience of using modules in Swift: what they are and +what they provide for the user. + +> **warning** +> +> This document was used in planning Swift 1.0; it has not been kept + +> up to date. + +High-Level Overview +=================== + +A module contains declarations +------------------------------ + +The primary purpose of a module is to provide declarations of types, +functions, and global variables that are present in a library. +Importing <import> the module gives access to these declarations +and allows them to be used in your code. + + import Chess + import Foundation + +You can also selectively import certain declarations from a module: + + import func Chess.createGreedyPlayer + import class Foundation.NSRegularExpression + +> **Comparison with Other Languages** +> +> Importing a module is much like importing a library in Ruby, Python, +> or Perl, importing a class in Java, or including a header file in a +> C-family language. However, unlike C, module files are not textually +> included and must be valid programs on their own, and may not be in a +> textual format at all. Unlike Java, declarations in a module are not +> visible at all until imported. And unlike the dynamic languages +> mentioned, importing a module cannot automatically cause any code to +> be run. + +Imported declarations can be accessed with qualified or unqualified lookup +-------------------------------------------------------------------------- + +Once a module has been imported, its declarations are available for use +within the current source file. These declarations can be referred to by +name, or by qualifying <qualified name> them with the name of the +module: + + func playChess(blackPlayer : Chess.Player, whitePlayer : Chess.Player) { + var board = Board() // refers to Chess.Board + } + +Modules provide a unique context for declarations +------------------------------------------------- + +A declaration in a module is unique; it is never the same as a +declaration with the same name in another module (with one caveat +described below). This means that two types `Chess.Board` and +`Xiangqi.Board` can exist in the same program, and each can be referred +to as `Board` as long as the other is not visible. If more than one +imported module declares the same name, the full qualified name can be +used for disambiguation. + +> **note** +> +> This is accomplished by including the module name in the mangled name +> of a declaration. Therefore, it is an ABI-breaking change to change +> the name of a module containing a public declaration. + +> **warning** +> +> The one exception to this rule is declarations that must be compatible +> with Objective-C. Such declarations follow the usual Objective-C rules +> for name conflicts: all classes must have unique names, all protocols +> must have unique names, and all constructors, methods, and properties +> must have unique names within their class (including inherited methods +> and properties). + +Modules may contain code +------------------------ + +In addition to declarations, modules may contain implementations of the +functions they define. The compiler may choose to use this information +when optimizing a user's program, usually by inlining the module code +into a caller. In some cases [^1], the compiler may even use a module's +function implementations to produce more effective diagnostics. + +Modules can also contain autolinking information, which the compiler +passes on to the linker. This can be used to specify which library +implements the declarations in the module. + +Modules can "re-export" other modules +------------------------------------- + +> **warning** +> +> This feature is likely to be modified in the future. + +Like any other body of code, a module may depend on other modules in its +implementation. The module implementer may also choose to re-export +these modules, meaning that anyone who imports the first module will +also have access to the declarations in the re-exported modules. : + + @exported import AmericanCheckers + +As an example, the "Cocoa" framework on OS X exists only to re-export +three other frameworks: AppKit, Foundation, and CoreData. + +Just as certain declarations can be selectively imported from a module, +so too can they be selectively re-exported, using the same syntax: + + @exported import class AmericanCheckers.Board + +Modules are uniquely identified by their name +--------------------------------------------- + +Module names exist in a global namespace and must be unique. Like type +names, module names are conventionally capitalized. + +> **TODO** +> +> While this matches the general convention for Clang, there are +> advantages to being able to rename a module for lookup purposes, even +> if the ABI name stays the same. It would also be nice to avoid having +> people stick prefixes on their module names the way they currently do +> for Objective-C classes. + +> **note** +> +> Because access into a module and access into a type look the same, it +> is bad style to declare a type with the same name as a top-level +> module used in your program: +> +> // Example 1: +> import Foundation +> import struct BuildingConstruction.Foundation +> +> var firstSupport = Foundation.SupportType() // from the struct or from the module? +> +> +> // Example 2: +> import Foundation +> import BuildingConstruction +> +> Foundation.SupportType() // from the class or from the module? +> +> In both cases, the type takes priority over the module, but this +> should still be avoided. +> +> > **TODO** +> > +> > Can we enforce this in the compiler? It seems like there's no way +> > around Example 2, and indeed Example 2 is probably doing the wrong +> > thing. + +`import` +======== + +As shown above, a module is imported using the `import` keyword, +followed by the name of the module: + + import AppKit + +To import only a certain declaration from the module, you use the +appropriate declaration keyword: + + import class AppKit.NSWindow + import func AppKit.NSApplicationMain + import var AppKit.NSAppKitVersionNumber + import typealias AppKit.NSApplicationPresentationOptions + +- `import typealias` has slightly special behavior: it will match any + type other than a protocol, regardless of how the type is declared + in the imported module. +- `import class`, `struct`, and `enum` will succeed even if the name + given is a typealias for a type of the appropriate kind. +- `import func` will bring in all overloads of the named function. +- Using a keyword that doesn't match the named declaration is + an error. + +> **TODO** +> +> There is currently no way to selectively import extensions or +> operators. + +Multiple source files +--------------------- + +Most programs are broken up into multiple source files, and these files +may depend on each other. To facilitate this design, declarations in +*all* source files in a module (including the "main module" for an +executable) are implicitly visible in each file's context. It is almost +as if all these files had been loaded with `import`, but with a few +important differences: + +- The declarations in other files belong to the module being built, + just like those in the current file. Therefore, if you need to refer + to them by qualified name, you need to use the name of the module + being built. +- A module is a fully-contained entity: it may depend on other + modules, but those other modules can't depend on it. Source files + within a module may have mutual dependencies. + +> **FIXME** +> +> This wouldn't belong in the user model at all except for the implicit +> visibility thing. Is there a better way to talk about this? + +Ambiguity +--------- + +Because two different modules can declare the same name, it is sometimes +necessary to use a qualified name to refer to a particular declaration: + + import Chess + import Xiangqi + + if userGame == "chess" { + Chess.playGame() + } else if userGame == "xiangqi" { + Xiangqi.playGame() + } + +Here, both modules declare a function named `playGame` that takes no +arguments, so we have to disambiguate by "qualifying" the function name +with the appropriate module. + +These are the rules for resolving name lookup ambiguities: + +1. Declarations in the current source file are best. +2. Declarations from other files in the same module are better than + declarations from imports. +3. Declarations from selective imports are better than declarations + from non-selective imports. (This may be used to give priority to a + particular module for a given name.) +4. Every source file implicitly imports the core standard library as a + non-selective import. +5. If the name refers to a function, normal overload resolution may + resolve ambiguities. + +Submodules +---------- + +> **warning** +> +> This feature was never implemented, or even fully designed. + +For large projects, it is usually desirable to break a single +application or framework into subsystems, which Swift calls +"submodules". A submodule is a development-time construct used for +grouping within a module. By default, declarations within a submodule +are considered "submodule-private", which means they are only visible +within that submodule (rather than across the entire module). These +declarations will not conflict with declarations in other submodules +that may have the same name. + +Declarations explicitly marked "whole-module" or "API" are still visible +across the entire module (even if declared within a submodule), and must +have a unique name within that space. + +The qualified name of a declaration within a submodule consists of the +top-level module name, followed by the submodule name, followed by the +declaration. + +> **note** +> +> Submodules are an opportunity feature for Swift 1.0. + +> **TODO** +> +> We need to decide once and for all whether implicit visibility applies +> across submodule boundaries, i.e. "can I access the public +> Swift.AST.Module from Swift.Sema without an import, or do I have to +> say `import Swift.AST`?" +> +> Advantages of module-wide implicit visibility: +> +> - Better name conflict checking. (The alternative is a linker error, +> or worse *no* linker error if the names have different manglings.) +> - Less work if things move around. +> - Build time performance is consistent whether or not you use +> this feature. +> +> Advantages of submodule-only implicit visibility: +> +> - Code completion will include names of public things you don't +> care about. +> - We haven't actually tested the build time performance of any large +> Swift projects, so we don't know if we can actually handle targets +> that contain hundreds of files. +> - Could be considered desirable to force declaring your internal +> dependencies explicitly. +> - In this mode, we could allow two "whole-module" declarations to +> have the same name, since they won't. (We could allow this in the +> other mode too but then the qualified name would always +> be required.) +> +> Both cases still use "submodule-only" as the default access control, +> so this only affects the implicit visibility of whole-module and +> public declarations. + +Import Search Paths +------------------- + +> **FIXME** +> +> Write this section. Can source files be self-contained modules? How +> does -i mode work? Can the "wrong" module be found when looking for a +> dependency (i.e. can I substitute my own Foundation and expect AppKit +> to work)? How are modules stored on disk? How do hierarchical module +> names work? + +Interoperability with Objective-C via Clang +=========================================== + +The compiler has the ability to interoperate with C and Objective-C by +importing Clang modules <Clang module>. This feature of the Clang +compiler was developed to provide a "semantic import" extension to the C +family of languages. The Swift compiler uses this to expose declarations +from C and Objective-C as if they used native Swift types. + +In all the examples above, `import AppKit` has been using this +mechanism: the module found with the name "AppKit" is generated from the +Objective-C AppKit framework. + +Clang Submodules +---------------- + +Clang also has a concept of "submodules", which are essentially +hierarchically-named modules. Unlike Swift's submodules, Clang +submodules are visible from outside the module. It is conventional for a +top-level Clang module to re-export all of its submodules, but sometimes +certain submodules are specified to require an explicit import: + + import OpenGL.GL3 + +Module Overlays +--------------- + +> **warning** +> +> This feature has mostly been removed from Swift; it's only in use + +> in the "overlay" libraries bundled with Swift itself. + +If a source file in module A includes `import A`, this indicates that +the source file is providing a replacement or overlay for an external +module. In most cases, the source file will re-export the underlying +module, but add some convenience APIs to make the existing interface +more Swift-friendly. + +This replacement syntax (using the current module name in an import) +cannot be used to overlay a Swift module, because module-naming. + +Multiple source files, part 2 +----------------------------- + +In migrating from Objective-C to Swift, it is expected that a single +program will contain a mix of sources. The compiler therefore allows +importing a single Objective-C header, exposing its declarations to the +main source file by constructing a sort of "ad hoc" module. These can +then be used like any other declarations imported from C or Objective-C. + +> **note** +> +> This is describing the feature that eventually became "bridging + +> headers" for app targets. + +Accessing Swift declarations from Objective-C +--------------------------------------------- + +> **warning** +> +> This never actually happened; instead, we went with "generated + +> headers" output by the Swift compiler. + +Using the new `@import` syntax, Objective-C translation units can import +Swift modules as well. Swift declarations will be mirrored into +Objective-C and can be called natively, just as Objective-C declarations +are mirrored into Swift for Clang modules <Clang module>. In this +case, only the declarations compatible with Objective-C will be visible. + +> **TODO** +> +> We need to actually do this, but it requires working on a branch of +> Clang, so we're pushing it back in the schedule as far as possible. +> The workaround is to manually write header files for imported Swift +> classes. + +> **TODO** +> +> Importing Swift sources from within the same target is a goal, but +> there are many difficulties. How do you name a file to be imported? +> What if the file itself depends on another Objective-C header? What if +> there's a mutual dependency across the language boundary? (That's a +> problem in both directions, since both Clang modules and Swift modules +> are only supposed to be exposed once they've been type-checked.) + +Glossary +======== + +[^1]: Specifically, code marked with the `@_transparent` attribute is + required to be "transparent" to the compiler: it *must* be inlined + and will affect diagnostics. diff --git a/docs/Modules.rst b/docs/Modules.rst deleted file mode 100644 index 21d0ae1855bdb..0000000000000 --- a/docs/Modules.rst +++ /dev/null @@ -1,472 +0,0 @@ -:orphan: - -.. @raise litre.TestsAreMissing -.. default-role:: term -.. title:: Modules User Model - - -A `module` is the primary unit of code sharing in Swift. This document -describes the experience of using modules in Swift: what they are and what they -provide for the user. - -.. warning:: This document was used in planning Swift 1.0; it has not been kept - up to date. - -.. contents:: :local: - - -High-Level Overview -=================== - -A module contains declarations ------------------------------- - -The primary purpose of a module is to provide declarations of types, functions, -and global variables that are present in a library. `Importing ` the -module gives access to these declarations and allows them to be used in your -code. - -:: - - import Chess - import Foundation - -You can also selectively import certain declarations from a module:: - - import func Chess.createGreedyPlayer - import class Foundation.NSRegularExpression - -.. admonition:: Comparison with Other Languages - - Importing a module is much like importing a library in Ruby, Python, or Perl, - importing a class in Java, or including a header file in a C-family language. - However, unlike C, module files are not textually included and must be valid - programs on their own, and may not be in a textual format at all. Unlike Java, - declarations in a module are not visible at all until imported. And unlike the - dynamic languages mentioned, importing a module cannot automatically cause - any code to be run. - - -Imported declarations can be accessed with qualified or unqualified lookup --------------------------------------------------------------------------- - - -Once a module has been imported, its declarations are available for use within -the current source file. These declarations can be referred to by name, or -by `qualifying ` them with the name of the module:: - - func playChess(blackPlayer : Chess.Player, whitePlayer : Chess.Player) { - var board = Board() // refers to Chess.Board - } - - -Modules provide a unique context for declarations -------------------------------------------------- - -A declaration in a module is unique; it is never the same as a declaration with -the same name in another module (with one caveat described below). This means -that two types ``Chess.Board`` and ``Xiangqi.Board`` can exist in the same -program, and each can be referred to as ``Board`` as long as the other is not -visible. If more than one imported module declares the same name, the full -`qualified name` can be used for disambiguation. - -.. note:: - - This is accomplished by including the module name in the `mangled name` of a - declaration. Therefore, it is an ABI-breaking change to change the name of a - module containing a public declaration. - -.. warning:: - - The one exception to this rule is declarations that must be compatible with - Objective-C. Such declarations follow the usual Objective-C rules for name - conflicts: all classes must have unique names, all protocols must have unique - names, and all constructors, methods, and properties must have unique names - within their class (including inherited methods and properties). - - -Modules may contain code ------------------------- - -In addition to declarations, modules may contain implementations of the -functions they define. The compiler may choose to use this information when -optimizing a user's program, usually by inlining the module code into a caller. -In some cases [#]_, the compiler may even use a module's function -implementations to produce more effective diagnostics. - -Modules can also contain `autolinking` information, which the compiler passes -on to the linker. This can be used to specify which library implements the -declarations in the module. - -.. [#] Specifically, code marked with the ``@_transparent`` attribute is - required to be "transparent" to the compiler: it *must* be inlined and - will affect diagnostics. - - -Modules can "re-export" other modules -------------------------------------- - -.. warning:: This feature is likely to be modified in the future. - -Like any other body of code, a module may depend on other modules in its -implementation. The module implementer may also choose to `re-export` these -modules, meaning that anyone who imports the first module will also have access -to the declarations in the re-exported modules. :: - - @exported import AmericanCheckers - -As an example, the "Cocoa" `framework` on OS X exists only to re-export three -other frameworks: AppKit, Foundation, and CoreData. - -Just as certain declarations can be selectively imported from a module, so too -can they be selectively re-exported, using the same syntax:: - - @exported import class AmericanCheckers.Board - - -.. _module-naming: - -Modules are uniquely identified by their name ---------------------------------------------- - -Module names exist in a global namespace and must be unique. Like type names, -module names are conventionally capitalized. - -.. admonition:: TODO - - While this matches the general convention for Clang, there are advantages to - being able to rename a module for lookup purposes, even if the ABI name stays - the same. It would also be nice to avoid having people stick prefixes on their - module names the way they currently do for Objective-C classes. - -.. note:: - - Because access into a module and access into a type look the same, it is bad - style to declare a type with the same name as a top-level module used in your - program:: - - // Example 1: - import Foundation - import struct BuildingConstruction.Foundation - - var firstSupport = Foundation.SupportType() // from the struct or from the module? - - - // Example 2: - import Foundation - import BuildingConstruction - - Foundation.SupportType() // from the class or from the module? - - In both cases, the type takes priority over the module, but this should still - be avoided. - - .. admonition:: TODO - - Can we enforce this in the compiler? It seems like there's no way around - Example 2, and indeed Example 2 is probably doing the wrong thing. - - -``import`` -========== - -As shown above, a module is imported using the ``import`` keyword, followed by -the name of the module:: - - import AppKit - -To import only a certain declaration from the module, you use the appropriate -declaration keyword:: - - import class AppKit.NSWindow - import func AppKit.NSApplicationMain - import var AppKit.NSAppKitVersionNumber - import typealias AppKit.NSApplicationPresentationOptions - -- ``import typealias`` has slightly special behavior: it will match any type - other than a protocol, regardless of how the type is declared in the imported - module. -- ``import class``, ``struct``, and ``enum`` will succeed even if the - name given is a typealias for a type of the appropriate kind. -- ``import func`` will bring in all overloads of the named function. -- Using a keyword that doesn't match the named declaration is an error. - -.. admonition:: TODO - - There is currently no way to selectively import extensions or operators. - - -.. _implicit-visibility: - -Multiple source files ---------------------- - -Most programs are broken up into multiple source files, and these files may -depend on each other. To facilitate this design, declarations in *all* source -files in a module (including the "main module" for an executable) are implicitly -visible in each file's context. It is almost as if all these files had been -loaded with ``import``, but with a few important differences: - -- The declarations in other files belong to the module being built, just like - those in the current file. Therefore, if you need to refer to them by - qualified name, you need to use the name of the module being built. -- A module is a fully-contained entity: it may depend on other modules, but - those other modules can't depend on it. Source files within a module may - have mutual dependencies. - -.. admonition:: FIXME - - This wouldn't belong in the user model at all except for the implicit - visibility thing. Is there a better way to talk about this? - - -Ambiguity ---------- - -Because two different modules can declare the same name, it is sometimes -necessary to use a `qualified name` to refer to a particular declaration:: - - import Chess - import Xiangqi - - if userGame == "chess" { - Chess.playGame() - } else if userGame == "xiangqi" { - Xiangqi.playGame() - } - -Here, both modules declare a function named ``playGame`` that takes no -arguments, so we have to disambiguate by "qualifying" the function name with -the appropriate module. - -These are the rules for resolving name lookup ambiguities: - -1. Declarations in the current source file are best. -2. Declarations from other files in the same module are better than - declarations from imports. -3. Declarations from selective imports are better than declarations from - non-selective imports. (This may be used to give priority to a particular - module for a given name.) -4. Every source file implicitly imports the core standard library as a - non-selective import. -5. If the name refers to a function, normal overload resolution may resolve - ambiguities. - -.. _submodules: - -Submodules ----------- - -.. warning:: This feature was never implemented, or even fully designed. - -For large projects, it is usually desirable to break a single application or -framework into subsystems, which Swift calls "submodules". A submodule is a -development-time construct used for grouping within a module. By default, -declarations within a submodule are considered "submodule-private", which -means they are only visible within that submodule (rather than across the -entire module). These declarations will not conflict with declarations in other -submodules that may have the same name. - -Declarations explicitly marked "whole-module" or "API" are still visible -across the entire module (even if declared within a submodule), and must have a -unique name within that space. - -The `qualified name` of a declaration within a submodule consists of the -top-level module name, followed by the submodule name, followed by the -declaration. - -.. note:: - - Submodules are an opportunity feature for Swift 1.0. - -.. admonition:: TODO - - We need to decide once and for all whether implicit visibility applies across - submodule boundaries, i.e. "can I access the public Swift.AST.Module from - Swift.Sema without an import, or do I have to say ``import Swift.AST``?" - - Advantages of module-wide implicit visibility: - - - Better name conflict checking. (The alternative is a linker error, or worse - *no* linker error if the names have different manglings.) - - Less work if things move around. - - Build time performance is consistent whether or not you use this feature. - - Advantages of submodule-only implicit visibility: - - - Code completion will include names of public things you don't care about. - - We haven't actually tested the build time performance of any large Swift - projects, so we don't know if we can actually handle targets that contain - hundreds of files. - - Could be considered desirable to force declaring your internal dependencies - explicitly. - - In this mode, we could allow two "whole-module" declarations to have the - same name, since they won't. (We could allow this in the other mode too - but then the qualified name would always be required.) - - Both cases still use "submodule-only" as the default access control, so this - only affects the implicit visibility of whole-module and public declarations. - - -Import Search Paths -------------------- - -.. admonition:: FIXME - - Write this section. Can source files be self-contained modules? How does -i - mode work? Can the "wrong" module be found when looking for a dependency - (i.e. can I substitute my own Foundation and expect AppKit to work)? - How are modules stored on disk? How do hierarchical module names work? - - -Interoperability with Objective-C via Clang -=========================================== - -The compiler has the ability to interoperate with C and Objective-C by -importing `Clang modules `. This feature of the Clang compiler -was developed to provide a "semantic import" extension to the C family of -languages. The Swift compiler uses this to expose declarations from C and -Objective-C as if they used native Swift types. - -In all the examples above, ``import AppKit`` has been using this mechanism: -the module found with the name "AppKit" is generated from the Objective-C -AppKit framework. - - -Clang Submodules ----------------- - -Clang also has a concept of "submodules", which are essentially hierarchically- -named modules. Unlike Swift's :ref:`submodules`, Clang submodules are visible -from outside the module. It is conventional for a top-level Clang module to -re-export all of its submodules, but sometimes certain submodules are specified -to require an explicit import:: - - import OpenGL.GL3 - - -Module Overlays ---------------- - -.. warning:: This feature has mostly been removed from Swift; it's only in use - in the "overlay" libraries bundled with Swift itself. - -If a source file in module A includes ``import A``, this indicates that the -source file is providing a replacement or overlay for an external module. -In most cases, the source file will `re-export` the underlying module, but -add some convenience APIs to make the existing interface more Swift-friendly. - -This replacement syntax (using the current module name in an import) cannot -be used to overlay a Swift module, because :ref:`module-naming`. - - -Multiple source files, part 2 ------------------------------ - -In migrating from Objective-C to Swift, it is expected that a single program -will contain a mix of sources. The compiler therefore allows importing a single -Objective-C header, exposing its declarations to the main source file by -constructing a sort of "ad hoc" module. These can then be used like any -other declarations imported from C or Objective-C. - -.. note:: This is describing the feature that eventually became "bridging - headers" for app targets. - - -Accessing Swift declarations from Objective-C ---------------------------------------------- - -.. warning:: This never actually happened; instead, we went with "generated - headers" output by the Swift compiler. - -Using the new ``@import`` syntax, Objective-C translation units can import -Swift modules as well. Swift declarations will be mirrored into Objective-C -and can be called natively, just as Objective-C declarations are mirrored into -Swift for `Clang modules `. In this case, only the declarations -compatible with Objective-C will be visible. - -.. admonition:: TODO - - We need to actually do this, but it requires working on a branch of Clang, so - we're pushing it back in the schedule as far as possible. The workaround is - to manually write header files for imported Swift classes. - -.. admonition:: TODO - - Importing Swift sources from within the same target is a goal, but there are - many difficulties. How do you name a file to be imported? What if the file - itself depends on another Objective-C header? What if there's a mutual - dependency across the language boundary? (That's a problem in both directions, - since both Clang modules and Swift modules are only supposed to be exposed - once they've been type-checked.) - - -Glossary -======== - -.. glossary:: - - autolinking - A technique where linking information is included in compiled object files, - so that external dependencies can be recorded without having to explicitly - specify them at link time. - - Clang module - A module whose contents are generated from a C-family header or set of - headers. See Clang's Modules__ documentation for more information. - - __ http://goto.apple.com/?http://clang.llvm.org/docs/Modules.html - - framework - A mechanism for library distribution on OS X. Traditionally contains header - files describing the library's API, a binary file containing the - implementation, and a directory containing any resources the library may - need. - - Frameworks are also used on iOS, but as of iOS 7 custom frameworks cannot - be created by users. - - import - To locate and read a module, then make its declarations available in the - current context. - - library - Abstractly, a collection of APIs for a programmer to use, usually with a - common theme. Concretely, the file containing the implementation of these - APIs. - - mangled name - A unique, internal name for a type or value. The term is most commonly used - in C++; see Wikipedia__ for some examples. Swift's name mangling scheme is - not the same as C++'s but serves a similar purpose. - - __ http://goto.apple.com/?http://en.wikipedia.org/wiki/Name_mangling#Name_mangling_in_C.2B.2B - - module - An entity containing the API for a library, to be `imported ` into - a source file. - - qualified name - A multi-piece name like ``Foundation.NSWindow``, which names an entity - within a particular context. This document is concerned with the case where - the context is the name of an imported module. - - re-export - To directly expose the API of one module through another module. Including - the latter module in a source file will behave as if the user had also - included the former module. - - serialized module - A particular encoding of a module that contains declarations that have - already been processed by the compiler. It may also contain implementations - of some function declarations in `SIL` form. - - SIL - "Swift Intermediate Language", a stable IR for the distribution of - inlineable code. - - - target - A dynamic library, framework, plug-in, or application to be built. - A natural LTO boundary, and roughly the same as what Xcode requires - separate targets to build. diff --git a/docs/MutationModel.md b/docs/MutationModel.md new file mode 100644 index 0000000000000..52913e064800b --- /dev/null +++ b/docs/MutationModel.md @@ -0,0 +1,181 @@ + +Immutability and Read-Only Methods +================================== + +Abstract + +: Swift programmers can already express the concept of read-only + properties and subscripts, and can express their intention to write + on a function parameter. However, the model is incomplete, which + currently leads to the compiler to accept (and silently drop) + mutations made by methods of these read-only entities. This proposal + completes the model, and additionally allows the user to declare + truly immutable data. + +The Problem +----------- + +Consider: + + class Window { + + var title: String { // title is not writable + get { + return somethingComputed() + } + } + } + + var w = Window() + w.title += " (parenthesized remark)” + +What do we do with this? Since `+=` has an `inout` first argument, we +detect this situation statically (hopefully one day we’ll have a better +error message): + + :1:9: error: expression does not type-check + w.title += " (parenthesized remark)" + ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Great. Now what about this? [^1] : + + w.title.append(" (fool the compiler)") + +Today, we allow it, but since there’s no way to implement the write-back +onto `w.title`, the changes are silently dropped. + +Unsatisfying Approaches +----------------------- + +We considered three alternatives to the current proposal, none of which +were considered satisfactory: + +1. Ban method calls on read-only properties of value type +2. Ban read-only properties of value type +3. Status quo: silently drop the effects of some method calls + +For rationales explaining why these approaches were rejected, please +refer to earlier versions of this document. + +Proposed Solution +----------------- + +### Terminology + +Classes and generic parameters that conform to a protocol attributed +`@class_protocol` are called **reference types**. All other types are +**value types**. + +### Mutating and Read-Only Methods + +A method attributed with `inout` is considered **mutating**. Otherwise, +it is considered **read-only**. + +The implicit `self` parameter of a struct or enum method is semantically +an `inout` parameter if and only if the method is attributed with +`mutating`. Read-only methods do not “write back” onto their target +objects. + +A program that applies the `mutating` to a method of a class—or of a +protocol attributed with `@class_protocol`—is ill-formed. \[Note: it is +logically consistent to think of all methods of classes as read-only, +even though they may in fact modify instance variables, because they +never “write back” onto the source reference.\] + +### Mutating Operations + +The following are considered **mutating operations** on an lvalue + +1. Assignment to the lvalue +2. Taking its address + +Remember that the following operations all take an lvalue's address +implicitly: + +- passing it to a mutating method: + + var x = Number(42) + x.increment() // mutating operation + +- passing it to a function attributed with `@assignment`: + + var y = 31 + y += 3 // mutating operation + +- assigning to a subscript or property (including an + instance variable) of a value type: + + x._i = 3 // mutating operation + var z: Array = [1000] + z[0] = 2 // mutating operation + +### Binding for Rvalues + +Just as `var` declares a name for an lvalue, `let` now gives a name to +an rvalue: + +The grammar rules for `let` are identical to those for `var`. + +### Properties and Subscripts + +A subscript or property access expression is an rvalue if + +- the property or subscript has no `set` clause +- the target of the property or subscript expression is an rvalue of + value type + +For example, consider this extension to our `Number` struct: + +Also imagine we have a class called `CNumber` defined exactly the same +way as `Number` (except that it's a class). Then, the following table +holds: + ++----------------------+----------------------------------+------------------------+ +| Declaration:|:: | | | | |:: | |Expression | var x = Number(42) // this +| | | | var x = CNumber(42) // or this | let x = Number(42) | | | let x += CNumber(42) // or this | | ++======================+==================================+========================+ +| `x.readOnlyValue` |**rvalue** (no `set` clause) |**rvalue** (target is +an| | | |rvalue of value type) | | | | | +----------------------+ | | | +`x[3]` | | | | | | | | | | | ++----------------------+----------------------------------+ | | +`x.writeableValue` |**lvalue** (has `set` clause) | | | | | | ++----------------------+ | | | `x["tree"]` | | | | | | | ++----------------------+----------------------------------+ | | `x.name` +|**lvalue** (instance variables | | | |implicitly have a `set` | | | +|clause) | | ++----------------------+----------------------------------+------------------------+ + +### The Big Rule + +> **error** +> +> A program that applies a mutating operation to an rvalue is ill-formed + +For example: + +### Non-`inout` Function Parameters are RValues + +A function that performs a mutating operation on a parameter is +ill-formed unless that parameter was marked with `inout`. A method that +performs a mutating operation on `self` is ill-formed unless the method +is attributed with `mutating`: + +### Protocols and Constraints + +When a protocol declares a property or `subscript` requirement, a +`{ get }` or `{ get set }` clause is always required. + +Where a `{ get set }` clause appears, the corresponding expression on a +type that conforms to the protocol must be an lvalue or the program is +ill-formed: + +------------------------------------------------------------------------ + +[^1]: String will acquire an `append(other: String)` method as part of + the formatting plan, but this scenario applies equally to any method + of a value type diff --git a/docs/MutationModel.rst b/docs/MutationModel.rst deleted file mode 100644 index dae65de7431d2..0000000000000 --- a/docs/MutationModel.rst +++ /dev/null @@ -1,301 +0,0 @@ -:orphan: - -.. raw:: html - - - - -================================== -Immutability and Read-Only Methods -================================== - -:Abstract: Swift programmers can already express the concept of - read-only properties and subscripts, and can express their - intention to write on a function parameter. However, the - model is incomplete, which currently leads to the compiler - to accept (and silently drop) mutations made by methods of - these read-only entities. This proposal completes the - model, and additionally allows the user to declare truly - immutable data. - -The Problem -=========== - -Consider:: - - class Window { - - var title: String { // title is not writable - get { - return somethingComputed() - } - } - } - - var w = Window() - w.title += " (parenthesized remark)” - -What do we do with this? Since ``+=`` has an ``inout`` first -argument, we detect this situation statically (hopefully one day we’ll -have a better error message): - -:: - - :1:9: error: expression does not type-check - w.title += " (parenthesized remark)" - ~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Great. Now what about this? [#append]_ :: - - w.title.append(" (fool the compiler)") - -Today, we allow it, but since there’s no way to implement the -write-back onto ``w.title``, the changes are silently dropped. - -Unsatisfying Approaches -======================= - -We considered three alternatives to the current proposal, none of -which were considered satisfactory: - -1. Ban method calls on read-only properties of value type -2. Ban read-only properties of value type -3. Status quo: silently drop the effects of some method calls - -For rationales explaining why these approaches were rejected, please -refer to earlier versions of this document. - -Proposed Solution -================= - -Terminology ------------ - -Classes and generic parameters that conform to a protocol attributed -``@class_protocol`` are called **reference types**. All other types -are **value types**. - -Mutating and Read-Only Methods ------------------------------- - -A method attributed with ``inout`` is considered **mutating**. -Otherwise, it is considered **read-only**. - -.. parsed-literal:: - - struct Number { - init(x: Int) { name = x.toString() } - - func getValue() { // read-only method - return Int(name) - } - **mutating** func increment() { // mutating method - name = (Int(name)+1).toString() - } - var name: String - } - -The implicit ``self`` parameter of a struct or enum method is semantically an -``inout`` parameter if and only if the method is attributed with -``mutating``. Read-only methods do not “write back” onto their target -objects. - -A program that applies the ``mutating`` to a method of a -class—or of a protocol attributed with ``@class_protocol``—is -ill-formed. [Note: it is logically consistent to think of all methods -of classes as read-only, even though they may in fact modify instance -variables, because they never “write back” onto the source reference.] - -Mutating Operations -------------------- - -The following are considered **mutating operations** on an lvalue - -1. Assignment to the lvalue -2. Taking its address - -Remember that the following operations all take an lvalue's address -implicitly: - -* passing it to a mutating method:: - - var x = Number(42) - x.increment() // mutating operation - -* passing it to a function attributed with ``@assignment``:: - - var y = 31 - y += 3 // mutating operation - -* assigning to a subscript or property (including an instance - variable) of a value type:: - - x._i = 3 // mutating operation - var z: Array = [1000] - z[0] = 2 // mutating operation - -Binding for Rvalues -------------------- - -Just as ``var`` declares a name for an lvalue, ``let`` now gives a -name to an rvalue: - -.. parsed-literal:: - - var clay = 42 - **let** stone = clay + 100 // stone can now be used as an rvalue - -The grammar rules for ``let`` are identical to those for ``var``. - -Properties and Subscripts -------------------------- - -A subscript or property access expression is an rvalue if - -* the property or subscript has no ``set`` clause -* the target of the property or subscript expression is an rvalue of - value type - -For example, consider this extension to our ``Number`` struct: - -.. parsed-literal:: - - extension Number { - var readOnlyValue: Int { return getValue() } - - var writableValue: Int { - get { - return getValue() - } - **set(x)** { - name = x.toString() - } - } - - subscript(n: Int) -> String { return name } - subscript(n: String) -> Int { - get { - return 42 - } - **set(x)** { - name = x.toString() - } - } - } - -Also imagine we have a class called ``CNumber`` defined exactly the -same way as ``Number`` (except that it's a class). Then, the -following table holds: - -+----------------------+----------------------------------+------------------------+ -| Declaration:|:: | | -| | |:: | -|Expression | var x = Number(42) // this | | -| | var x = CNumber(42) // or this | let x = Number(42) | -| | let x = CNumber(42) // or this | | -+======================+==================================+========================+ -| ``x.readOnlyValue`` |**rvalue** (no ``set`` clause) |**rvalue** (target is an| -| | |rvalue of value type) | -| | | | -+----------------------+ | | -| ``x[3]`` | | | -| | | | -| | | | -+----------------------+----------------------------------+ | -| ``x.writeableValue`` |**lvalue** (has ``set`` clause) | | -| | | | -+----------------------+ | | -| ``x["tree"]`` | | | -| | | | -+----------------------+----------------------------------+ | -| ``x.name`` |**lvalue** (instance variables | | -| |implicitly have a ``set`` | | -| |clause) | | -+----------------------+----------------------------------+------------------------+ - -The Big Rule -------------- - -.. Error:: A program that applies a mutating operation to an rvalue is ill-formed - :class: warning - -For example: - -.. parsed-literal:: - - clay = 43 // OK; a var is always assignable - **stone =** clay \* 1000 // **Error:** stone is an rvalue - - swap(&clay, **&stone**) // **Error:** 'stone' is an rvalue; can't take its address - - **stone +=** 3 // **Error:** += is declared inout, @assignment and thus - // implicitly takes the address of 'stone' - - **let** x = Number(42) // x is an rvalue - x.getValue() // ok, read-only method - x.increment() // **Error:** calling mutating method on rvalue - x.readOnlyValue // ok, read-only property - x.writableValue // ok, there's no assignment to writableValue - x.writableValue++ // **Error:** assigning into a property of an immutable value - -Non-``inout`` Function Parameters are RValues ----------------------------------------------- - -A function that performs a mutating operation on a parameter is -ill-formed unless that parameter was marked with ``inout``. A -method that performs a mutating operation on ``self`` is ill-formed -unless the method is attributed with ``mutating``: - -.. parsed-literal:: - - func f(x: Int, inout y: Int) { - y = x // ok, y is an inout parameter - x = y // **Error:** function parameter 'x' is immutable - } - -Protocols and Constraints -------------------------- - -When a protocol declares a property or ``subscript`` requirement, a -``{ get }`` or ``{ get set }`` clause is always required. - -.. parsed-literal:: - - protocol Bitset { - var count: Int { **get** } - var intValue: Int { **get set** } - subscript(bitIndex: Int) -> Bool { **get set** } - } - -Where a ``{ get set }`` clause appears, the corresponding expression -on a type that conforms to the protocol must be an lvalue or the -program is ill-formed: - -.. parsed-literal:: - - struct BS { - var count: Int // ok; an lvalue or an rvalue is fine - - var intValue : Int { - get { - return 3 - } - set { // ok, lvalue required and has a set clause - ignore(value) - } - } - - subscript(i: Int) -> Bool { - return true // **Error:** needs a 'set' clause to yield an lvalue - } - } - ------------------ - -.. [#append] String will acquire an ``append(other: String)`` method as part of the - formatting plan, but this scenario applies equally to any - method of a value type diff --git a/docs/ObjectInitialization.md b/docs/ObjectInitialization.md new file mode 100644 index 0000000000000..2d5e6f4a8bd22 --- /dev/null +++ b/docs/ObjectInitialization.md @@ -0,0 +1,582 @@ +Object Initialization +===================== + +> **warning** +> +> This document is incomplete and not up-to-date; it currently + +> describes the initialization model from Swift 1.0. + +Introduction +------------ + +*Object initialization* is the process by which a new object is +allocated, its stored properties initialized, and any additional setup +tasks are performed, including allowing its superclass's to perform +their own initialization. *Object teardown* is the reverse process, +performing teardown tasks, destroying stored properties, and eventually +deallocating the object. + +Initializers +------------ + +An initializer is responsible for the initialization of an object. +Initializers are introduced with the `init` keyword. For example: + + class A { + var i: Int + var s: String + + init int(i: Int) string(s: String) { + self.i = i + self.s = s + completeInit() + } + + func completeInit() { /* ... */ } + } + +Here, the class `A` has an initializer that accepts an `Int` and a +`String`, and uses them to initialize its two stored properties, then +calls another method to perform other initialization tasks. The +initializer can be invoked by constructing a new `A` object: + + var a = A(int: 17, string: "Seventeen") + +The allocation of the new `A` object is implicit in the construction +syntax, and cannot be separated from the call to the initializer. + +Within an initializer, all of the stored properties must be initialized +(via an assignment) before `self` can be used in any way. For example, +the following would produce a compiler error: + + init int(i: Int) string(s: String) { + completeInit() // error: variable 'self.i' used before being initialized + self.i = i + self.s = s + } + +A stored property with an initial value declared within the class is +considered to be initialized at the beginning of the initializer. For +example, the following is a valid initializer: + + class A2 { + var i: Int = 17 + var s: String = "Seventeen" + + init int(i: Int) string(s: String) { + // okay: i and s are both initialized in the class + completeInit() + } + + func completeInit() { /* ... */ } + } + +After all stored properties have been initialized, one is free to use +`self` in any manner. + +### Designated Initializers + +There are two kinds of initializers in Swift: designated initializers +and convenience initializers. A *designated initializer* is responsible +for the primary initialization of an object, including the +initialization of any stored properties, *chaining* to one of its +superclass's designated initializers via a `super.init` call (if there +is a superclass), and performing any other initialization tasks, in that +order. For example, consider a subclass `B` of `A`: + + class B : A { + var d: Double + + init int(i: Int) string(s: String) { + self.d = Double(i) // initialize stored properties + super.init(int: i, string: s) // chain to superclass + completeInitForB() // perform other tasks + } + + func completeInitForB() { /* ... */ } + } + +Consider the following construction of an object of type `B`: + + var b = B(int: 17, string: "Seventeen") + +> **Note** +> +> Swift differs from many other languages in that it requires one to +> initialize stored properties *before* chaining to the superclass +> initializer. This is part of Swift's memory safety guarantee, and is +> discussed further in the section on Three-Phase +> Initialization\_. + +Initialization proceeds in several steps: + +1. An object of type `B` is allocated by the runtime. +2. `B`'s initializer initializes the stored property `d` to `17.0`. +3. `B`'s initializer chains to `A`'s initializer. +4. `A`'s initializer initialize's the stored properties `i` and `s`'. +5. `A`'s initializer calls `completeInit()`, then returns. +6. `B`'s initializer calls `completeInitForB()`, then returns. + +A class generally has a small number of designated initializers, which +act as funnel points through which the object will be initialized. All +of the designated initializers for a class must be written within the +class definition itself, rather than in an extension, because the +complete set of designated initializers is part of the interface +contract with subclasses of a class. + +The other, non-designated initializers of a class are called convenience +initializers, which tend to provide additional initialization +capabilities that are often more convenient for common tasks. + +### Convenience Initializers + +A *convenience initializer* is an initializer that provides an +alternative interface to the designated initializers of a class. A +convenience initializer is denoted by the return type `Self` in the +definition. Unlike designated initializers, convenience initializers can +be defined either in the class definition itself or within an extension +of the class. For example: + + extension A { + init() -> Self { + self.init(int: 17, string: "Seventeen") + } + } + +A convenience initializer cannot initialize the stored properties of the +class directly, nor can it invoke a superclass initializer via +`super.init`. Rather, it must *dispatch* to another initializer using +`self.init`, which is then responsible for initializing the object. A +convenience initializer is not permitted to access `self` (or anything +that depends on `self`, such as one of its properties) prior to the +`self.init` call, although it may freely access `self` after +`self.init`. + +Convenience initializers and designated initializers can both be used to +construct objects, using the same syntax. For example, the `A` +initializer above can be used to build a new `A` object without any +arguments: + + var a2 = A() // uses convenience initializer + +### Initializer Inheritance + +One of the primary benefits of convenience initializers is that they can +be inherited by subclasses. Initializer inheritance eliminates the need +to repeat common initialization code---such as initial values of stored +properties not easily written in the class itself, or common +registration tasks that occur during initialization---while using the +same initialization syntax. For example, this allows a `B` object to be +constructed with no arguments by using the inherited convenience +initializer defined in the previous section: + + var b2 = B() + +Initialization proceeds as follows: + +1. A `B` object is allocated by the runtime. +2. `A`'s convenience initializer `init()` is invoked. +3. `A`'s convenience initializer dispatches to `init int:string:` via + the `self.init` call. This call dynamically resolves to `B`'s + designated initializer. +4. `B`'s designated initializer initializes the stored property `d` to + `17.0`. +5. `B`'s designated initializer chains to `A`'s designated initializer. +6. `A`'s designated initializer initialize's the stored properties `i` + and `s`'. +7. `A`'s designated initializer calls `completeInit()`, then returns. +8. `B`'s designated initializer calls `completeInitForB()`, + then returns. +9. `A`'s convenience initializer returns. + +Convenience initializers are only inherited under certain circumstances. +Specifically, for a given subclass to inherit the convenience +initializers of its superclass, the subclass must override each of the +designated initializers of its superclass. For example `B` provides the +initializer `init int:string:`, which overrides `A`'s designated +initializer `init int:string:` because the initializer name and +parameters are the same. If we had some other subclass `OtherB` of `A` +that did not provide such an override, it would not inherit `A`'s +convenience initializers: + + class OtherB : A { + var d: Double + + init int(i: Int) string(s: String) double(d: Double) { + self.d = d // initialize stored properties + super.init(int: i, string: s) // chain to superclass + } + } + + var ob = OtherB() // error: A's convenience initializer init() not inherited + +> **Note** +> +> The requirement that a subclass override all of the designated +> initializers of its superclass to enable initializer inheritance is +> crucial to Swift's memory safety model. See Initializer +> Inheritance Model\_ for more information. + +Note that a subclass may have different designated initializers from its +superclass. This can occur in a number of ways. For example, the +subclass might override one of its superclass's designated initializers +with a convenience initializer: + + class YetAnotherB : A { + var d: Double + + init int(i: Int) string(s: String) -> Self { + self.init(int: i, string: s, double: Double(i)) // dispatch + } + + init int(i: Int) string(s: String) double(d: Double) { + self.d = d // initialize stored properties + super.init(int: i, string: s) // chain to superclass + } + } + + var yab = YetAnotherB() // okay: YetAnotherB overrides all of A's designated initializers + +In other cases, it's possible that the convenience initializers of the +superclass simply can't be made to work, because the subclass +initializers require additional information provided via a parameter +that isn't present in the convenience initializers of the superclass: + + class PickyB : A { + var notEasy: NoEasyDefault + + init int(i: Int) string(s: String) notEasy(NoEasyDefault) { + self.notEasy = notEasy + super.init(int: i, string: s) // chain to superclass + } + } + +Here, `PickyB` has a stored property of a type `NoEasyDefault` that +can't easily be given a default value: it has to be provided as a +parameter to one of `PickyB`'s initializers. Therefore, `PickyB` takes +over responsibility for its own initialization, and none of `A`'s +convenience initializers will be inherited into `PickyB`. + +### Synthesized Initializers + +When a particular class does not specify any designated initializers, +the implementation will synthesize initializers for the class when all +of the class's stored properties have initial values in the class. The +form of the synthesized initializers depends on the superclass (if +present). + +When a superclass is present, the compiler synthesizes a new designated +initializer in the subclass for each designated initializer of the +superclass. For example, consider the following class `C`: + + class C : B { + var title: String = "Default Title" + } + +The superclass `B` has a single designated initializer,: + + init int(i: Int) string(s: String) + +Therefore, the compiler synthesizes the following designated initializer +in `C`, which chains to the corresponding designated initializer in the +superclass: + + init int(i: Int) string(s: String) { + // title is already initialized in the class C + super.init(int: i, string: s) + } + +The result of this synthesis is that all designated initializers of the +superclass are (automatically) overridden in the subclass, becoming +designated initializers of the subclass as well. Therefore, any +convenience initializers in the superclass are also inherited, allowing +the subclass (`C`) to be constructed with the same initializers as the +superclass (`B`): + + var c1 = C(int: 17, string: "Seventeen") + var c2 = C() + +When the class has no superclass, a default initializer (with no +parameters) is implicitly defined: + + class D { + var title = "Default Title" + + /* implicitly defined */ + init() { } + } + + var d = D() // uses implicitly-defined default initializer + +### Required Initializers + +Objects are generally constructed with the construction syntax `T(...)` +used in all of the examples above, where `T` is the name of the type. +However, it is occasionally useful to construct an object for which the +actual type is not known until runtime. For example, one might have a +`View` class that expects to be initialized with a specific set of +coordinates: + + struct Rect { + var origin: (Int, Int) + var dimensions: (Int, Int) + } + + class View { + init frame(Rect) { /* initialize view */ } + } + +The actual initialization of a subclass of `View` would then be +performed at runtime, with the actual subclass being determined via some +external file that describes the user interface. The actual +instantiation of the object would use a type value: + + func createView(viewClass: View.Type, frame: Rect) -> View { + return viewClass(frame: frame) // error: 'init frame:' is not 'required' + } + +The code above is invalid because there is no guarantee that a given +subclass of `View` will have an initializer `init frame:`, because the +subclass might have taken over its own initialization (as with `PickyB`, +above). To require that all subclasses provide a particular initializer, +use the `required` attribute as follows: + + class View { + @required init frame(Rect) { + /* initialize view */ + } + } + + func createView(viewClass: View.Type, frame: Rect) -> View { + return viewClass(frame: frame) // okay + } + +The `required` attribute allows the initializer to be used to construct +an object of a dynamically-determined subclass, as in the `createView` +method. It places the (transitive) requirement on all subclasses of +`View` to provide an initializer `init frame:`. For example, the +following `Button` subclass would produce an error: + + class Button : View { + // error: 'Button' does not provide required initializer 'init frame:'. + } + +The fix is to implement the required initializer in `Button`: + + class Button : View { + @required init frame(Rect) { // okay: satisfies requirement + super.init(frame: frame) + } + } + +### Initializers in Protocols + +Initializers may be declared within a protocol. For example: + + protocol DefaultInitializable { + init() + } + +> **Note** +> +> Initializers in protocols have not yet been implemented. Stay tuned. + +A class can satisfy this requirement by providing a required +initializer. For example, only the first of the two following classes +conforms to its protocol: + + class DefInit : DefaultInitializable { + @required init() { } + } + + class AlmostDefInit : DefaultInitializable { + init() { } // error: initializer used for protocol conformance must be 'required' + } + +The `required` requirement ensures that all subclasses of the class +declaring conformance to the protocol will also have the initializer, so +they too will conform to the protocol. This allows one to construct +objects given type values of protocol type: + + func createAnyDefInit(typeVal: DefaultInitializable.Type) -> DefaultInitializable { + return typeVal() + } + +### De-initializers + +While initializers are responsible for setting up an object's state, +*de-initializers* are responsible for tearing down that state. Most +classes don't require a de-initializer, because Swift automatically +releases all stored properties and calls to the superclass's +de-initializer. However, if your class has allocated a resource that is +not an object (say, a Unix file descriptor) or has registered itself +during initialization, one can write a de-initializer using `deinit`: + + class FileHandle { + var fd: Int32 + + init withFileDescriptor(fd: Int32) { + self.fd = fd + } + + deinit { + close(fd) + } + } + +The statements within a de-initializer (here, the call to `close`) +execute first, then the superclass's de-initializer is called. Finally, +stored properties are released and the object is deallocated. + +### Methods Returning `Self` + +A class method can have the special return type `Self`, which refers to +the dynamic type of `self`. Such a method guarantees that it will return +an object with the same dynamic type as `self`. One of the primary uses +of the `Self` return type is for factory methods: + + extension View { + class func createView(frame: Rect) -> Self { + return self(frame: frame) + } + } + +> **Note** +> +> The return type `Self` fulfills the same role as Objective-C's +> `instancetype`, although Swift provides stronger type checking for +> these methods. + +Within the body of this class method, the implicit parameter `self` is a +value with type `View.Type`, i.e., it's a type value for the class +`View` or one of its subclasses. Therefore, the restrictions are the +same as for any value of type `View.Type`: one can call other class +methods and construct new objects using required initializers of the +class, among other things. The result returned from such a method must +be derived from the type of `Self`. For example, it cannot return a +value of type `View`, because `self` might refer to some subclass of +`View`. + +Instance methods can also return `Self`. This is typically used to allow +chaining of method calls by returning `Self` from each method, as in the +builder pattern: + + class DialogBuilder { + func setTitle(title: String) -> Self { + // set the title + return self; + } + + func setBounds(frame: Rect) -> Self { + // set the bounds + return self; + } + } + + var builder = DialogBuilder() + .setTitle("Hello, World!") + .setBounds(Rect(0, 0, 640, 480)) + +Memory Safety +------------- + +Swift aims to provide memory safety by default, and much of the design +of Swift's object initialization scheme is in service of that goal. This +section describes the rationale for the design based on the +memory-safety goals of the language. + +### Three-Phase Initialization + +The three-phase initialization model used by Swift's initializers +ensures that all stored properties get initialized before any code can +make use of `self`. This is important uses of `self`---say, calling a +method on `self`---could end up referring to stored properties before +they are initialized. Consider the following Objective-C code, where +instance variables are initialized *after* the call to the superclass +initializer: + + @interface A : NSObject + - (instancetype)init; + - (void)finishInit; + @end + + @implementation A + - (instancetype)init { + self = [super init]; + if (self) { + [self finishInit]; + } + return self; + } + @end + + @interface B : A + @end + + @implementation B { + NSString *ivar; + } + + - (instancetype)init { + self = [super init]; + if (self) { + self->ivar = @"Default name"; + } + return self; + } + + - (void) finishInit { + NSLog(@"ivar has the value %@\n", self->ivar); + } + @end + +> **Notes** +> +> In Objective-C, `+alloc` zero-initializes all of the instance +> variables, which gives them predictable behavior before the init +> method gets to initialize them. Given that Objective-C is fairly +> resilient to `nil` objects, this default behavior eliminates (or +> hides) many such initialization bugs. In Swift, however, the +> zero-initialized state is less likely to be valid, and the memory +> safety goals are stronger, so zero-initialization does not suffice. + +When initializing a `B` object, the `NSLog` statement will print: + + ivar has the value (null) + +because `-[B finishInit]` executes before `B` has had a chance to +initialize its instance variables. Swift initializers avoid this issue +by splitting each initializer into three phases: + +1\. Initialize stored properties. In this phase, the compiler verifies +that `self` is not used except when writing to the stored properties of +the current class (not its superclasses!). Additionally, this +initialization directly writes to the storage of the stored properties, +and does not call any setter or `willSet`/`didSet` method. In this +phase, it is not possible to read any of the stored properties. + +2\. Call to superclass initializer, if any. As with the first step, +`self` cannot be accessed at all. + +3\. Perform any additional initialization tasks, which may call methods +on `self`, access properties, and so on. + +Note that, with this scheme, `self` cannot be used until the original +class and all of its superclasses have initialized their stored +properties, closing the memory safety hole. + +### Initializer Inheritance Model + +FIXME: To be written + +Objective-C Interoperability +---------------------------- + +### Initializers and Init Methods + +### Designated and Convenience Initializers + +### Allocation and Deallocation + +### Dynamic Subclassing diff --git a/docs/ObjectInitialization.rst b/docs/ObjectInitialization.rst deleted file mode 100644 index ed4ef873d105f..0000000000000 --- a/docs/ObjectInitialization.rst +++ /dev/null @@ -1,609 +0,0 @@ -Object Initialization -===================== - -.. warning:: This document is incomplete and not up-to-date; it currently - describes the initialization model from Swift 1.0. - -.. contents:: - -Introduction ------------- - -*Object initialization* is the process by which a new object is -allocated, its stored properties initialized, and any additional setup -tasks are performed, including allowing its superclass's to perform -their own initialization. *Object teardown* is the reverse process, -performing teardown tasks, destroying stored properties, and -eventually deallocating the object. - -Initializers ------------- - -An initializer is responsible for the initialization of an -object. Initializers are introduced with the ``init`` keyword. For -example:: - - class A { - var i: Int - var s: String - - init int(i: Int) string(s: String) { - self.i = i - self.s = s - completeInit() - } - - func completeInit() { /* ... */ } - } - -Here, the class ``A`` has an initializer that accepts an ``Int`` and a -``String``, and uses them to initialize its two stored properties, -then calls another method to perform other initialization tasks. The -initializer can be invoked by constructing a new ``A`` object:: - - var a = A(int: 17, string: "Seventeen") - -The allocation of the new ``A`` object is implicit in the -construction syntax, and cannot be separated from the call to the -initializer. - -Within an initializer, all of the stored properties must be -initialized (via an assignment) before ``self`` can be used in any -way. For example, the following would produce a compiler error:: - - init int(i: Int) string(s: String) { - completeInit() // error: variable 'self.i' used before being initialized - self.i = i - self.s = s - } - -A stored property with an initial value declared within the class is -considered to be initialized at the beginning of the initializer. For -example, the following is a valid initializer:: - - class A2 { - var i: Int = 17 - var s: String = "Seventeen" - - init int(i: Int) string(s: String) { - // okay: i and s are both initialized in the class - completeInit() - } - - func completeInit() { /* ... */ } - } - -After all stored properties have been initialized, one is free to use -``self`` in any manner. - -Designated Initializers -~~~~~~~~~~~~~~~~~~~~~~~ - -There are two kinds of initializers in Swift: designated initializers -and convenience initializers. A *designated initializer* is -responsible for the primary initialization of an object, including the -initialization of any stored properties, *chaining* to one of its -superclass's designated initializers via a ``super.init`` call (if -there is a superclass), and performing any other initialization tasks, -in that order. For example, consider a subclass ``B`` of ``A``:: - - class B : A { - var d: Double - - init int(i: Int) string(s: String) { - self.d = Double(i) // initialize stored properties - super.init(int: i, string: s) // chain to superclass - completeInitForB() // perform other tasks - } - - func completeInitForB() { /* ... */ } - } - -Consider the following construction of an object of type ``B``:: - - var b = B(int: 17, string: "Seventeen") - -.. sidebar:: Note - - Swift differs from many other languages in that it requires one to - initialize stored properties *before* chaining to the superclass - initializer. This is part of Swift's memory safety guarantee, and - is discussed further in the section on `Three-Phase - Initialization`_. - -Initialization proceeds in several steps: - -1. An object of type ``B`` is allocated by the runtime. -2. ``B``'s initializer initializes the stored property ``d`` to - ``17.0``. -3. ``B``'s initializer chains to ``A``'s initializer. -4. ``A``'s initializer initialize's the stored properties ``i`` and - ``s``'. -5. ``A``'s initializer calls ``completeInit()``, then returns. -6. ``B``'s initializer calls ``completeInitForB()``, then returns. - -A class generally has a small number of designated initializers, which -act as funnel points through which the object will be -initialized. All of the designated initializers for a class must be -written within the class definition itself, rather than in an -extension, because the complete set of designated initializers is part -of the interface contract with subclasses of a class. - -The other, non-designated initializers of a class are called -convenience initializers, which tend to provide additional -initialization capabilities that are often more convenient for common -tasks. - -Convenience Initializers -~~~~~~~~~~~~~~~~~~~~~~~~ - -A *convenience initializer* is an initializer that provides an -alternative interface to the designated initializers of a class. A -convenience initializer is denoted by the return type ``Self`` in the -definition. Unlike designated initializers, convenience initializers -can be defined either in the class definition itself or within an -extension of the class. For example:: - - extension A { - init() -> Self { - self.init(int: 17, string: "Seventeen") - } - } - -A convenience initializer cannot initialize the stored properties of -the class directly, nor can it invoke a superclass initializer via -``super.init``. Rather, it must *dispatch* to another initializer -using ``self.init``, which is then responsible for initializing the -object. A convenience initializer is not permitted to access ``self`` -(or anything that depends on ``self``, such as one of its properties) -prior to the ``self.init`` call, although it may freely access -``self`` after ``self.init``. - -Convenience initializers and designated initializers can both be used -to construct objects, using the same syntax. For example, the ``A`` -initializer above can be used to build a new ``A`` object without any -arguments:: - - var a2 = A() // uses convenience initializer - -Initializer Inheritance -~~~~~~~~~~~~~~~~~~~~~~~ - -One of the primary benefits of convenience initializers is that they -can be inherited by subclasses. Initializer inheritance eliminates the -need to repeat common initialization code---such as initial values of -stored properties not easily written in the class itself, or common -registration tasks that occur during initialization---while using the -same initialization syntax. For example, this allows a ``B`` object to -be constructed with no arguments by using the inherited convenience -initializer defined in the previous section:: - - var b2 = B() - -Initialization proceeds as follows: - -1. A ``B`` object is allocated by the runtime. -2. ``A``'s convenience initializer ``init()`` is invoked. -3. ``A``'s convenience initializer dispatches to ``init int:string:`` - via the ``self.init`` call. This call dynamically resolves to - ``B``'s designated initializer. -4. ``B``'s designated initializer initializes the stored property - ``d`` to ``17.0``. -5. ``B``'s designated initializer chains to ``A``'s designated - initializer. -6. ``A``'s designated initializer initialize's the stored properties - ``i`` and ``s``'. -7. ``A``'s designated initializer calls ``completeInit()``, then - returns. -8. ``B``'s designated initializer calls ``completeInitForB()``, then - returns. -9. ``A``'s convenience initializer returns. - -Convenience initializers are only inherited under certain -circumstances. Specifically, for a given subclass to inherit the -convenience initializers of its superclass, the subclass must override -each of the designated initializers of its superclass. For example -``B`` provides the initializer ``init int:string:``, which overrides -``A``'s designated initializer ``init int:string:`` because the -initializer name and parameters are the same. If we had some other -subclass ``OtherB`` of ``A`` that did not provide such an override, it -would not inherit ``A``'s convenience initializers:: - - class OtherB : A { - var d: Double - - init int(i: Int) string(s: String) double(d: Double) { - self.d = d // initialize stored properties - super.init(int: i, string: s) // chain to superclass - } - } - - var ob = OtherB() // error: A's convenience initializer init() not inherited - -.. sidebar:: Note - - The requirement that a subclass override all of the designated - initializers of its superclass to enable initializer inheritance is - crucial to Swift's memory safety model. See `Initializer - Inheritance Model`_ for more information. - -Note that a subclass may have different designated initializers from -its superclass. This can occur in a number of ways. For example, the -subclass might override one of its superclass's designated -initializers with a convenience initializer:: - - class YetAnotherB : A { - var d: Double - - init int(i: Int) string(s: String) -> Self { - self.init(int: i, string: s, double: Double(i)) // dispatch - } - - init int(i: Int) string(s: String) double(d: Double) { - self.d = d // initialize stored properties - super.init(int: i, string: s) // chain to superclass - } - } - - var yab = YetAnotherB() // okay: YetAnotherB overrides all of A's designated initializers - -In other cases, it's possible that the convenience initializers of the -superclass simply can't be made to work, because the subclass -initializers require additional information provided via a -parameter that isn't present in the convenience initializers of the -superclass:: - - class PickyB : A { - var notEasy: NoEasyDefault - - init int(i: Int) string(s: String) notEasy(NoEasyDefault) { - self.notEasy = notEasy - super.init(int: i, string: s) // chain to superclass - } - } - -Here, ``PickyB`` has a stored property of a type ``NoEasyDefault`` -that can't easily be given a default value: it has to be provided as a -parameter to one of ``PickyB``'s initializers. Therefore, ``PickyB`` -takes over responsibility for its own initialization, and -none of ``A``'s convenience initializers will be inherited into -``PickyB``. - -Synthesized Initializers -~~~~~~~~~~~~~~~~~~~~~~~~ - -When a particular class does not specify any designated initializers, -the implementation will synthesize initializers for the class when all -of the class's stored properties have initial values in the class. The -form of the synthesized initializers depends on the superclass (if -present). - -When a superclass is present, the compiler synthesizes a new -designated initializer in the subclass for each designated initializer -of the superclass. For example, consider the following class ``C``:: - - class C : B { - var title: String = "Default Title" - } - -The superclass ``B`` has a single designated initializer,:: - - init int(i: Int) string(s: String) - -Therefore, the compiler synthesizes the following designated -initializer in ``C``, which chains to the corresponding designated -initializer in the superclass:: - - init int(i: Int) string(s: String) { - // title is already initialized in the class C - super.init(int: i, string: s) - } - -The result of this synthesis is that all designated initializers of -the superclass are (automatically) overridden in the subclass, -becoming designated initializers of the subclass as well. Therefore, -any convenience initializers in the superclass are also inherited, -allowing the subclass (``C``) to be constructed with the same -initializers as the superclass (``B``):: - - var c1 = C(int: 17, string: "Seventeen") - var c2 = C() - -When the class has no superclass, a default initializer (with no -parameters) is implicitly defined:: - - class D { - var title = "Default Title" - - /* implicitly defined */ - init() { } - } - - var d = D() // uses implicitly-defined default initializer - -Required Initializers -~~~~~~~~~~~~~~~~~~~~~ - -Objects are generally constructed with the construction syntax -``T(...)`` used in all of the examples above, where ``T`` is the name -of the type. However, it is occasionally useful to construct an object -for which the actual type is not known until runtime. For example, one -might have a ``View`` class that expects to be initialized with a -specific set of coordinates:: - - struct Rect { - var origin: (Int, Int) - var dimensions: (Int, Int) - } - - class View { - init frame(Rect) { /* initialize view */ } - } - -The actual initialization of a subclass of ``View`` would then be -performed at runtime, with the actual subclass being determined via -some external file that describes the user interface. The actual -instantiation of the object would use a type value:: - - func createView(viewClass: View.Type, frame: Rect) -> View { - return viewClass(frame: frame) // error: 'init frame:' is not 'required' - } - -The code above is invalid because there is no guarantee that a given -subclass of ``View`` will have an initializer ``init frame:``, because -the subclass might have taken over its own initialization (as with -``PickyB``, above). To require that all subclasses provide a -particular initializer, use the ``required`` attribute as follows:: - - class View { - @required init frame(Rect) { - /* initialize view */ - } - } - - func createView(viewClass: View.Type, frame: Rect) -> View { - return viewClass(frame: frame) // okay - } - -The ``required`` attribute allows the initializer to be used to -construct an object of a dynamically-determined subclass, as in the -``createView`` method. It places the (transitive) requirement on all -subclasses of ``View`` to provide an initializer ``init frame:``. For -example, the following ``Button`` subclass would produce an error:: - - class Button : View { - // error: 'Button' does not provide required initializer 'init frame:'. - } - -The fix is to implement the required initializer in ``Button``:: - - class Button : View { - @required init frame(Rect) { // okay: satisfies requirement - super.init(frame: frame) - } - } - -Initializers in Protocols -~~~~~~~~~~~~~~~~~~~~~~~~~ - -Initializers may be declared within a protocol. For example:: - - protocol DefaultInitializable { - init() - } - -.. sidebar:: Note - - Initializers in protocols have not yet been implemented. Stay tuned. - -A class can satisfy this requirement by providing a required -initializer. For example, only the first of the two following classes -conforms to its protocol:: - - class DefInit : DefaultInitializable { - @required init() { } - } - - class AlmostDefInit : DefaultInitializable { - init() { } // error: initializer used for protocol conformance must be 'required' - } - -The ``required`` requirement ensures that all subclasses of the class -declaring conformance to the protocol will also have the initializer, -so they too will conform to the protocol. This allows one to construct -objects given type values of protocol type:: - - func createAnyDefInit(typeVal: DefaultInitializable.Type) -> DefaultInitializable { - return typeVal() - } - -De-initializers -~~~~~~~~~~~~~~~ - -While initializers are responsible for setting up an object's state, -*de-initializers* are responsible for tearing down that state. Most -classes don't require a de-initializer, because Swift automatically -releases all stored properties and calls to the superclass's -de-initializer. However, if your class has allocated a resource that -is not an object (say, a Unix file descriptor) or has registered -itself during initialization, one can write a de-initializer using -``deinit``:: - - class FileHandle { - var fd: Int32 - - init withFileDescriptor(fd: Int32) { - self.fd = fd - } - - deinit { - close(fd) - } - } - -The statements within a de-initializer (here, the call to ``close``) -execute first, then the superclass's de-initializer is -called. Finally, stored properties are released and the object is -deallocated. - -Methods Returning ``Self`` -~~~~~~~~~~~~~~~~~~~~~~~~~~ - - -A class method can have the special return type ``Self``, which refers -to the dynamic type of ``self``. Such a method guarantees that it will -return an object with the same dynamic type as ``self``. One of the -primary uses of the ``Self`` return type is for factory methods:: - - extension View { - class func createView(frame: Rect) -> Self { - return self(frame: frame) - } - } - -.. sidebar:: Note - - The return type ``Self`` fulfills the same role as Objective-C's - ``instancetype``, although Swift provides stronger type checking for - these methods. - -Within the body of this class method, the implicit parameter ``self`` -is a value with type ``View.Type``, i.e., it's a type value for the -class ``View`` or one of its subclasses. Therefore, the restrictions -are the same as for any value of type ``View.Type``: one can call -other class methods and construct new objects using required -initializers of the class, among other things. The result returned -from such a method must be derived from the type of ``Self``. For -example, it cannot return a value of type ``View``, because ``self`` -might refer to some subclass of ``View``. - -Instance methods can also return ``Self``. This is typically used to -allow chaining of method calls by returning ``Self`` from each method, -as in the builder pattern:: - - class DialogBuilder { - func setTitle(title: String) -> Self { - // set the title - return self; - } - - func setBounds(frame: Rect) -> Self { - // set the bounds - return self; - } - } - - var builder = DialogBuilder() - .setTitle("Hello, World!") - .setBounds(Rect(0, 0, 640, 480)) - - -Memory Safety -------------- - -Swift aims to provide memory safety by default, and much of the design -of Swift's object initialization scheme is in service of that -goal. This section describes the rationale for the design based on the -memory-safety goals of the language. - -Three-Phase Initialization -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The three-phase initialization model used by Swift's initializers -ensures that all stored properties get initialized before any code can -make use of ``self``. This is important uses of ``self``---say, -calling a method on ``self``---could end up referring to stored -properties before they are initialized. Consider the following -Objective-C code, where instance variables are initialized *after* the -call to the superclass initializer:: - - @interface A : NSObject - - (instancetype)init; - - (void)finishInit; - @end - - @implementation A - - (instancetype)init { - self = [super init]; - if (self) { - [self finishInit]; - } - return self; - } - @end - - @interface B : A - @end - - @implementation B { - NSString *ivar; - } - - - (instancetype)init { - self = [super init]; - if (self) { - self->ivar = @"Default name"; - } - return self; - } - - - (void) finishInit { - NSLog(@"ivar has the value %@\n", self->ivar); - } - @end - -.. sidebar:: Notes - - In Objective-C, ``+alloc`` zero-initializes all of the instance - variables, which gives them predictable behavior before the init - method gets to initialize them. Given that Objective-C is fairly - resilient to ``nil`` objects, this default behavior eliminates (or - hides) many such initialization bugs. In Swift, however, the - zero-initialized state is less likely to be valid, and the memory - safety goals are stronger, so zero-initialization does not suffice. - -When initializing a ``B`` object, the ``NSLog`` statement will print:: - - ivar has the value (null) - -because ``-[B finishInit]`` executes before ``B`` has had a chance to -initialize its instance variables. Swift initializers avoid this issue -by splitting each initializer into three phases: - -1. Initialize stored properties. In this phase, the compiler verifies -that ``self`` is not used except when writing to the stored properties -of the current class (not its superclasses!). Additionally, this -initialization directly writes to the storage of the stored -properties, and does not call any setter or ``willSet``/``didSet`` -method. In this phase, it is not possible to read any of the stored -properties. - -2. Call to superclass initializer, if any. As with the first step, -``self`` cannot be accessed at all. - -3. Perform any additional initialization tasks, which may call methods -on ``self``, access properties, and so on. - -Note that, with this scheme, ``self`` cannot be used until the -original class and all of its superclasses have initialized their -stored properties, closing the memory safety hole. - - -Initializer Inheritance Model -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -FIXME: To be written - -Objective-C Interoperability ----------------------------- - -Initializers and Init Methods -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Designated and Convenience Initializers -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Allocation and Deallocation -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Dynamic Subclassing -~~~~~~~~~~~~~~~~~~~ diff --git a/docs/OptimizationTips.md b/docs/OptimizationTips.md new file mode 100644 index 0000000000000..e044a745a92ee --- /dev/null +++ b/docs/OptimizationTips.md @@ -0,0 +1,545 @@ +Writing High-Performance Swift Code +=================================== + +The following document is a gathering of various tips and tricks for +writing high-performance Swift code. The intended audience of this +document is compiler and standard library developers. + +Some of the tips in this document can help improve the quality of your +Swift program and make your code less error prone and more readable. +Explicitly marking final-classes and class-protocols are two obvious +examples. However some of the tips described in this document are +unprincipled, twisted and come to solve a specific temporary limitation +of the compiler or the language. Many of the recommendations in this +document come with trade offs for things like program runtime, binary +size, code readability, etc. + +Enabling Optimizations +====================== + +The first thing one should always do is to enable optimization. Swift +provides three different optimization levels: + +- `-Onone`: This is meant for normal development. It performs minimal + optimizations and preserves all debug info. +- `-O`: This is meant for most production code. The compiler performs + aggressive optimizations that can drastically change the type and + amount of emitted code. Debug information will be emitted but will + be lossy. +- `-Ounchecked`: This is a special optimization mode meant for + specific libraries or applications where one is willing to trade + safety for performance. The compiler will remove all overflow checks + as well as some implicit type checks. This is not intended to be + used in general since it may result in undetected memory safety + issues and integer overflows. Only use this if you have carefully + reviewed that your code is safe with respect to integer overflow and + type casts. + +In the Xcode UI, one can modify the current optimization level as +follows: + +... + +Whole Module Optimizations +========================== + +By default Swift compiles each file individually. This allows Xcode to +compile multiple files in parallel very quickly. However, compiling each +file separately prevents certain compiler optimizations. Swift can also +compile the entire program as if it were one file and optimize the +program as if it were a single compilation unit. This mode is enabled +using the command line flag `-whole-module-optimization`. Programs that +are compiled in this mode will most likely take longer to compile, but +may run faster. + +This mode can be enabled using the Xcode build setting 'Whole Module +Optimization'. + +Reducing Dynamic Dispatch +========================= + +Swift by default is a very dynamic language like Objective-C. Unlike +Objective C, Swift gives the programmer the ability to improve runtime +performance when necessary by removing or reducing this dynamicism. This +section goes through several examples of language constructs that can be +used to perform such an operation. + +Dynamic Dispatch +---------------- + +Classes use dynamic dispatch for methods and property accesses by +default. Thus in the following code snippet, `a.aProperty`, +`a.doSomething()` and `a.doSomethingElse()` will all be invoked via +dynamic dispatch: + + class A { + var aProperty: [Int] + func doSomething() { ... } + dynamic doSomethingElse() { ... } + } + + class B : A { + override var aProperty { + get { ... } + set { ... } + } + + override func doSomething() { ... } + } + + func usingAnA(a: A) { + a.doSomething() + a.aProperty = ... + } + +In Swift, dynamic dispatch defaults to indirect invocation through a +vtable [^1]. If one attaches the `dynamic` keyword to the declaration, +Swift will emit calls via Objective-C message send instead. In both +cases this is slower than a direct function call because it prevents +many compiler optimizations [^2] in addition to the overhead of +performing the indirect call itself. In performance critical code, one +often will want to restrict this dynamic behavior. + +Advice: Use 'final' when you know the declaration does not need to be overridden +-------------------------------------------------------------------------------- + +The `final` keyword is a restriction on a declaration of a class, a +method, or a property such that the declaration cannot be overridden. +This implies that the compiler can emit direct function calls instead of +indirect calls. For instance in the following `C.array1` and `D.array1` +will be accessed directly [^3]. In contrast, `D.array2` will be called +via a vtable: + + final class C { + // No declarations in class 'C' can be overridden. + var array1: [Int] + func doSomething() { ... } + } + + class D { + final var array1 [Int] // 'array1' cannot be overridden by a computed property. + var array2: [Int] // 'array2' *can* be overridden by a computed property. + } + + func usingC(c: C) { + c.array1[i] = ... // Can directly access C.array without going through dynamic dispatch. + c.doSomething() = ... // Can directly call C.doSomething without going through virtual dispatch. + } + + func usingD(d: D) { + d.array1[i] = ... // Can directly access D.array1 without going through dynamic dispatch. + d.array2[i] = ... // Will access D.array2 through dynamic dispatch. + } + +Advice: Use 'private' when declaration does not need to be accessed outside of file +----------------------------------------------------------------------------------- + +Applying the `private` keyword to a declaration restricts the visibility +of the declaration to the file in which it is declared. This allows the +compiler to be able to ascertain all other potentially overriding +declarations. Thus the absence of any such declarations enables the +compiler to infer the `final` keyword automatically and remove indirect +calls for methods and field accesses accordingly. For instance in the +following, `e.doSomething()` and `f.myPrivateVar`, will be able to be +accessed directly assuming `E`, `F` do not have any overriding +declarations in the same file: + + private class E { + func doSomething() { ... } + } + + class F { + private var myPrivateVar : Int + } + + func usingE(e: E) { + e.doSomething() // There is no sub class in the file that declares this class. + // The compiler can remove virtual calls to doSomething() + // and directly call A’s doSomething method. + } + + func usingF(f: F) -> Int { + return f.myPrivateVar + } + +Using Container Types Efficiently +================================= + +An important feature provided by the Swift standard library are the +generic containers Array and Dictionary. This section will explain how +to use these types in a performant manner. + +Advice: Use value types in Array +-------------------------------- + +In Swift, types can be divided into two different categories: value +types (structs, enums, tuples) and reference types (classes). A key +distinction is that value types can not be included inside an NSArray. +Thus when using value types, the optimizer can remove most of the +overhead in Array that is necessary to handle the possibility of the +array being backed an NSArray. + +Additionally, In contrast to reference types, value types only need +reference counting if they contain, recursively, a reference type. By +using value types without reference types, one can avoid additional +retain, release traffic inside Array. + + // Don't use a class here. + struct PhonebookEntry { + var name : String + var number : [Int] + } + + var a : [PhonebookEntry] + +Keep in mind that there is a trade-off between using large value types +and using reference types. In certain cases, the overhead of copying and +moving around large value types will outweigh the cost of removing the +bridging and retain/release overhead. + +Advice: Use ContiguousArray with reference types when NSArray bridging is unnecessary +------------------------------------------------------------------------------------- + +If you need an array of reference types and the array does not need to +be bridged to NSArray, use ContiguousArray instead of Array: + + class C { ... } + var a: ContiguousArray = [C(...), C(...), ..., C(...)] + +Advice: Use inplace mutation instead of object-reassignment +----------------------------------------------------------- + +All standard library containers in Swift are value types that use COW +(copy-on-write) [^4] to perform copies instead of explicit copies. In +many cases this allows the compiler to elide unnecessary copies by +retaining the container instead of performing a deep copy. This is done +by only copying the underlying container if the reference count of the +container is greater than 1 and the container is mutated. For instance +in the following, no copying will occur when `d` is assigned to `c`, but +when `d` undergoes structural mutation by appending `2`, `d` will be +copied and then `2` will be appended to `d`: + + var c: [Int] = [ ... ] + var d = c // No copy will occur here. + d.append(2) // A copy *does* occur here. + +Sometimes COW can introduce additional unexpected copies if the user is +not careful. An example of this is attempting to perform mutation via +object-reassignment in functions. In Swift, all parameters are passed in +at +1, i.e. the parameters are retained before a callsite, and then are +released at the end of the callee. This means that if one writes a +function like the following: + + func append_one(a: [Int]) -> [Int] { + a.append(1) + return a + } + + var a = [1, 2, 3] + a = append_one(a) + +`a` may be copied [^5] despite the version of `a` without one appended +to it has no uses after `append_one` due to the assignment. This can be +avoided through the usage of `inout` parameters: + + func append_one_in_place(inout a: [Int]) { + a.append(1) + } + + var a = [1, 2, 3] + append_one_in_place(&a) + +Unchecked operations +==================== + +Swift eliminates integer overflow bugs by checking for overflow when +performing normal arithmetic. These checks are not appropriate in high +performance code where one knows that no memory safety issues can +result. + +Advice: Use unchecked integer arithmetic when you can prove that overflow can not occur +--------------------------------------------------------------------------------------- + +In performance-critical code you can elide overflow checks if you know +it is safe. + + a : [Int] + b : [Int] + c : [Int] + + // Precondition: for all a[i], b[i]: a[i] + b[i] does not overflow! + for i in 0 ... n { + c[i] = a[i] &+ b[i] + } + +Generics +======== + +Swift provides a very powerful abstraction mechanism through the use of +generic types. The Swift compiler emits one block of concrete code that +can perform `MySwiftFunc` for any `T`. The generated code takes a +table of function pointers and a box containing `T` as additional +parameters. Any differences in behavior between `MySwiftFunc` and +`MySwiftFunc` are accounted for by passing a different table of +function pointers and the size abstraction provided by the box. An +example of generics: + + class MySwiftFunc { ... } + + MySwiftFunc X // Will emit code that works with Int... + MySwiftFunc Y // ... as well as String. + +When optimizations are enabled, the Swift compiler looks at each +invocation of such code and attempts to ascertain the concrete (i.e. +non-generic type) used in the invocation. If the generic function's +definition is visible to the optimizer and the concrete type is known, +the Swift compiler will emit a version of the generic function +specialized to the specific type. This process, called *specialization*, +enables the removal of the overhead associated with generics. Some more +examples of generics: + + class MyStack { + func push(element: T) { ... } + func pop() -> T { ... } + } + + func myAlgorithm(a: [T], length: Int) { ... } + + // The compiler can specialize code of MyStack[Int] + var stackOfInts: MyStack[Int] + // Use stack of ints. + for i in ... { + stack.push(...) + stack.pop(...) + } + + var arrayOfInts: [Int] + // The compiler can emit a specialized version of 'myAlgorithm' targeted for + // [Int]' types. + myAlgorithm(arrayOfInts, arrayOfInts.length) + +Advice: Put generic declarations in the same file where they are used +--------------------------------------------------------------------- + +The optimizer can only perform specializations if the definition of the +generic declaration is visible in the current Module. This can only +occur if the declaration is in the same file as the invocation of the +generic. *NOTE* The standard library is a special case. Definitions in +the standard library are visible in all modules and available for +specialization. + +Advice: Allow the compiler to perform generic specialization +------------------------------------------------------------ + +The compiler can only specialize generic code if the call site and the +callee function are located in the same compilation unit. One trick that +we can use to allow compiler to optimize the callee function is to write +code that performs a type check in the same compilation unit as the +callee function. The code behind the type check then re-dispatches the +call to the generic function - but this time it has the type +information. In the code sample below we've inserted a type check into +the function "play\_a\_game" and made the code run hundreds of times +faster. + + //Framework.swift: + + protocol Pingable { func ping() -> Self } + protocol Playable { func play() } + + extension Int : Pingable { + func ping() -> Int { return self + 1 } + } + + class Game : Playable { + var t : T + + init (_ v : T) {t = v} + + func play() { + for _ in 0...100_000_000 { t = t.ping() } + } + } + + func play_a_game(game : Playable ) { + // This check allows the optimizer to specialize the + // generic call 'play' + if let z = game as? Game { + z.play() + } else { + game.play() + } + } + + /// -------------- >8 + + // Application.swift: + + play_a_game(Game(10)) + +The cost of large swift values +============================== + +In Swift, values keep a unique copy of their data. There are several +advantages to using value-types, like ensuring that values have +independent state. When we copy values (the effect of assignment, +initialization, and argument passing) the program will create a new copy +of the value. For some large values these copies could be time consuming +and hurt the performance of the program. + +Consider the example below that defines a tree using "value" nodes. The +tree nodes contain other nodes using a protocol. In computer graphics +scenes are often composed from different entities and transformations +that can be represented as values, so this example is somewhat +realistic. + + protocol P {} + struct Node : P { + var left, right : P? + } + + struct Tree { + var node : P? + init() { ... } + } + +When a tree is copied (passed as an argument, initialized or assigned) +the whole tree needs to be copied. In the case of our tree this is an +expensive operation that requires many calls to malloc/free and a +significant reference counting overhead. + +However, we don't really care if the value is copied in memory as long +as the semantics of the value remains. + +Advice: Use copy-on-write semantics for large values +---------------------------------------------------- + +To eliminate the cost of copying large values adopt copy-on-write +behavior. The easiest way to implement copy-on-write is to compose +existing copy-on-write data structures, such as Array. Swift arrays are +values, but the content of the array is not copied around every time the +array is passed as an argument because it features copy-on-write traits. + +In our Tree example we eliminate the cost of copying the content of the +tree by wrapping it in an array. This simple change has a major impact +on the performance of our tree data structure, and the cost of passing +the array as an argument drops from being O(n), depending on the size of +the tree to O(1). + + struct tree : P { + var node : [P?] + init() { + node = [ thing ] + } + } + +There are two obvious disadvantages of using Array for COW semantics. +The first problem is that Array exposes methods like "append" and +"count" that don't make any sense in the context of a value wrapper. +These methods can make the use of the reference wrapper awkward. It is +possible to work around this problem by creating a wrapper struct that +will hide the unused APIs and the optimizer will remove this overhead, +but this wrapper will not solve the second problem. The Second problem +is that Array has code for ensuring program safety and interaction with +Objective-C. Swift checks if indexed accesses fall within the array +bounds and when storing a value if the array storage needs to be +extended. These runtime checks can slow things down. + +An alternative to using Array is to implement a dedicated copy-on-write +data structure to replace Array as the value wrapper. The example below +shows how to construct such a data structure: + + final class Ref { + var val : T + init(_ v : T) {val = v} + } + + struct Box { + var ref : Ref + init(_ x : T) { ref = Ref(x) } + + var value: T { + get { return ref.val } + set { + if (!isUniquelyReferencedNonObjC(&ref)) { + ref = Ref(newValue) + return + } + ref.val = newValue + } + } + } + +The type `Box` can replace the array in the code sample above. + +Unsafe code +=========== + +Swift classes are always reference counted. The swift compiler inserts +code that increments the reference count every time the object is +accessed. For example, consider the problem of scanning a linked list +that's implemented using classes. Scanning the list is done by moving a +reference from one node to the next: `elem = elem.next`. Every time we +move the reference swift will increment the reference count of the +`next` object and decrement the reference count of the previous object. +These reference count operations are expensive and unavoidable when +using Swift classes. + + final class Node { + var next: Node? + var data: Int + ... + } + +Advice: Use unmanaged references to avoid reference counting overhead +--------------------------------------------------------------------- + +In performance-critical code you can use choose to use unmanaged +references. The `Unmanaged` structure allows developers to disable +automatic reference counting for a specific reference. + + var Ref : Unmanaged = Unmanaged.passUnretained(Head) + + while let Next = Ref.takeUnretainedValue().next { + ... + Ref = Unmanaged.passUnretained(Next) + } + +Protocols +========= + +Advice: Mark protocols that are only satisfied by classes as class-protocols +---------------------------------------------------------------------------- + +Swift can limit protocols adoption to classes only. One advantage of +marking protocols as class-only is that the compiler can optimize the +program based on the knowledge that only classes satisfy a protocol. For +example, the ARC memory management system can easily retain (increase +the reference count of an object) if it knows that it is dealing with a +class. Without this knowledge the compiler has to assume that a struct +may satisfy the protocol and it needs to be prepared to retain or +release non-trivial structures, which can be expensive. + +If it makes sense to limit the adoption of protocols to classes then +mark protocols as class-only protocols to get better runtime +performance. + + protocol Pingable : class { func ping() -> Int } + +Footnotes +========= + +[^1]: A virtual method table or 'vtable' is a type specific table + referenced by instances that contains the addresses of the type's + methods. Dynamic dispatch proceeds by first looking up the table + from the object and then looking up the method in the table. + +[^2]: This is due to the compiler not knowing the exact function being + called. + +[^3]: i.e. a direct load of a class's field or a direct call to a + function. + +[^4]: Explain what COW is here. + +[^5]: In certain cases the optimizer is able to via inlining and ARC + optimization remove the retain, release causing no copy to occur. diff --git a/docs/OptimizationTips.rst b/docs/OptimizationTips.rst deleted file mode 100644 index a8c66e01eb8fc..0000000000000 --- a/docs/OptimizationTips.rst +++ /dev/null @@ -1,582 +0,0 @@ -:orphan: - -Writing High-Performance Swift Code -=================================== - -The following document is a gathering of various tips and tricks for writing -high-performance Swift code. The intended audience of this document is compiler -and standard library developers. - -Some of the tips in this document can help improve the quality of your Swift -program and make your code less error prone and more readable. Explicitly -marking final-classes and class-protocols are two obvious examples. However some -of the tips described in this document are unprincipled, twisted and come to -solve a specific temporary limitation of the compiler or the language. Many of -the recommendations in this document come with trade offs for things like -program runtime, binary size, code readability, etc. - - -Enabling Optimizations -====================== - -The first thing one should always do is to enable optimization. Swift provides -three different optimization levels: - -- ``-Onone``: This is meant for normal development. It performs minimal - optimizations and preserves all debug info. -- ``-O``: This is meant for most production code. The compiler performs - aggressive optimizations that can drastically change the type and amount of - emitted code. Debug information will be emitted but will be lossy. -- ``-Ounchecked``: This is a special optimization mode meant for specific - libraries or applications where one is willing to trade safety for - performance. The compiler will remove all overflow checks as well as some - implicit type checks. This is not intended to be used in general since it may - result in undetected memory safety issues and integer overflows. Only use this - if you have carefully reviewed that your code is safe with respect to integer - overflow and type casts. - -In the Xcode UI, one can modify the current optimization level as follows: - -... - - -Whole Module Optimizations -========================== - -By default Swift compiles each file individually. This allows Xcode to -compile multiple files in parallel very quickly. However, compiling each file -separately prevents certain compiler optimizations. Swift can also compile -the entire program as if it were one file and optimize the program as if it -were a single compilation unit. This mode is enabled using the command -line flag ``-whole-module-optimization``. Programs that are compiled in -this mode will most likely take longer to compile, but may run faster. - -This mode can be enabled using the Xcode build setting 'Whole Module Optimization'. - - -Reducing Dynamic Dispatch -========================= - -Swift by default is a very dynamic language like Objective-C. Unlike Objective -C, Swift gives the programmer the ability to improve runtime performance when -necessary by removing or reducing this dynamicism. This section goes through -several examples of language constructs that can be used to perform such an -operation. - -Dynamic Dispatch ----------------- - -Classes use dynamic dispatch for methods and property accesses by default. Thus -in the following code snippet, ``a.aProperty``, ``a.doSomething()`` and -``a.doSomethingElse()`` will all be invoked via dynamic dispatch: - -:: - - class A { - var aProperty: [Int] - func doSomething() { ... } - dynamic doSomethingElse() { ... } - } - - class B : A { - override var aProperty { - get { ... } - set { ... } - } - - override func doSomething() { ... } - } - - func usingAnA(a: A) { - a.doSomething() - a.aProperty = ... - } - -In Swift, dynamic dispatch defaults to indirect invocation through a vtable -[#]_. If one attaches the ``dynamic`` keyword to the declaration, Swift will -emit calls via Objective-C message send instead. In both cases this is slower -than a direct function call because it prevents many compiler optimizations [#]_ -in addition to the overhead of performing the indirect call itself. In -performance critical code, one often will want to restrict this dynamic -behavior. - -Advice: Use 'final' when you know the declaration does not need to be overridden --------------------------------------------------------------------------------- - -The ``final`` keyword is a restriction on a declaration of a class, a method, or -a property such that the declaration cannot be overridden. This implies that the -compiler can emit direct function calls instead of indirect calls. For instance -in the following ``C.array1`` and ``D.array1`` will be accessed directly -[#]_. In contrast, ``D.array2`` will be called via a vtable: - -:: - - final class C { - // No declarations in class 'C' can be overridden. - var array1: [Int] - func doSomething() { ... } - } - - class D { - final var array1 [Int] // 'array1' cannot be overridden by a computed property. - var array2: [Int] // 'array2' *can* be overridden by a computed property. - } - - func usingC(c: C) { - c.array1[i] = ... // Can directly access C.array without going through dynamic dispatch. - c.doSomething() = ... // Can directly call C.doSomething without going through virtual dispatch. - } - - func usingD(d: D) { - d.array1[i] = ... // Can directly access D.array1 without going through dynamic dispatch. - d.array2[i] = ... // Will access D.array2 through dynamic dispatch. - } - -Advice: Use 'private' when declaration does not need to be accessed outside of file ------------------------------------------------------------------------------------ - -Applying the ``private`` keyword to a declaration restricts the visibility of -the declaration to the file in which it is declared. This allows the compiler to -be able to ascertain all other potentially overriding declarations. Thus the -absence of any such declarations enables the compiler to infer the ``final`` -keyword automatically and remove indirect calls for methods and field accesses -accordingly. For instance in the following, ``e.doSomething()`` and -``f.myPrivateVar``, will be able to be accessed directly assuming ``E``, ``F`` -do not have any overriding declarations in the same file: - -:: - - private class E { - func doSomething() { ... } - } - - class F { - private var myPrivateVar : Int - } - - func usingE(e: E) { - e.doSomething() // There is no sub class in the file that declares this class. - // The compiler can remove virtual calls to doSomething() - // and directly call A’s doSomething method. - } - - func usingF(f: F) -> Int { - return f.myPrivateVar - } - -Using Container Types Efficiently -================================= - -An important feature provided by the Swift standard library are the generic -containers Array and Dictionary. This section will explain how to use these -types in a performant manner. - -Advice: Use value types in Array --------------------------------- - -In Swift, types can be divided into two different categories: value types -(structs, enums, tuples) and reference types (classes). A key distinction is -that value types can not be included inside an NSArray. Thus when using value -types, the optimizer can remove most of the overhead in Array that is necessary -to handle the possibility of the array being backed an NSArray. - -Additionally, In contrast to reference types, value types only need reference -counting if they contain, recursively, a reference type. By using value types -without reference types, one can avoid additional retain, release traffic inside -Array. - -:: - - // Don't use a class here. - struct PhonebookEntry { - var name : String - var number : [Int] - } - - var a : [PhonebookEntry] - -Keep in mind that there is a trade-off between using large value types and using -reference types. In certain cases, the overhead of copying and moving around -large value types will outweigh the cost of removing the bridging and -retain/release overhead. - -Advice: Use ContiguousArray with reference types when NSArray bridging is unnecessary -------------------------------------------------------------------------------------- - -If you need an array of reference types and the array does not need to be -bridged to NSArray, use ContiguousArray instead of Array: - -:: - - class C { ... } - var a: ContiguousArray = [C(...), C(...), ..., C(...)] - -Advice: Use inplace mutation instead of object-reassignment ------------------------------------------------------------ - -All standard library containers in Swift are value types that use COW -(copy-on-write) [#]_ to perform copies instead of explicit copies. In many cases -this allows the compiler to elide unnecessary copies by retaining the container -instead of performing a deep copy. This is done by only copying the underlying -container if the reference count of the container is greater than 1 and the -container is mutated. For instance in the following, no copying will occur when -``d`` is assigned to ``c``, but when ``d`` undergoes structural mutation by -appending ``2``, ``d`` will be copied and then ``2`` will be appended to ``d``: - -:: - - var c: [Int] = [ ... ] - var d = c // No copy will occur here. - d.append(2) // A copy *does* occur here. - -Sometimes COW can introduce additional unexpected copies if the user is not -careful. An example of this is attempting to perform mutation via -object-reassignment in functions. In Swift, all parameters are passed in at +1, -i.e. the parameters are retained before a callsite, and then are released at the -end of the callee. This means that if one writes a function like the following: - -:: - - func append_one(a: [Int]) -> [Int] { - a.append(1) - return a - } - - var a = [1, 2, 3] - a = append_one(a) - -``a`` may be copied [#]_ despite the version of ``a`` without one appended to it -has no uses after ``append_one`` due to the assignment. This can be avoided -through the usage of ``inout`` parameters: - -:: - - func append_one_in_place(inout a: [Int]) { - a.append(1) - } - - var a = [1, 2, 3] - append_one_in_place(&a) - -Unchecked operations -==================== - -Swift eliminates integer overflow bugs by checking for overflow when performing -normal arithmetic. These checks are not appropriate in high performance code -where one knows that no memory safety issues can result. - -Advice: Use unchecked integer arithmetic when you can prove that overflow can not occur ---------------------------------------------------------------------------------------- - -In performance-critical code you can elide overflow checks if you know it is -safe. - -:: - - a : [Int] - b : [Int] - c : [Int] - - // Precondition: for all a[i], b[i]: a[i] + b[i] does not overflow! - for i in 0 ... n { - c[i] = a[i] &+ b[i] - } - -Generics -======== - -Swift provides a very powerful abstraction mechanism through the use of generic -types. The Swift compiler emits one block of concrete code that can perform -``MySwiftFunc`` for any ``T``. The generated code takes a table of function -pointers and a box containing ``T`` as additional parameters. Any differences in -behavior between ``MySwiftFunc`` and ``MySwiftFunc`` are accounted -for by passing a different table of function pointers and the size abstraction -provided by the box. An example of generics: - -:: - - class MySwiftFunc { ... } - - MySwiftFunc X // Will emit code that works with Int... - MySwiftFunc Y // ... as well as String. - -When optimizations are enabled, the Swift compiler looks at each invocation of -such code and attempts to ascertain the concrete (i.e. non-generic type) used in -the invocation. If the generic function's definition is visible to the optimizer -and the concrete type is known, the Swift compiler will emit a version of the -generic function specialized to the specific type. This process, called -*specialization*, enables the removal of the overhead associated with -generics. Some more examples of generics: - -:: - - class MyStack { - func push(element: T) { ... } - func pop() -> T { ... } - } - - func myAlgorithm(a: [T], length: Int) { ... } - - // The compiler can specialize code of MyStack[Int] - var stackOfInts: MyStack[Int] - // Use stack of ints. - for i in ... { - stack.push(...) - stack.pop(...) - } - - var arrayOfInts: [Int] - // The compiler can emit a specialized version of 'myAlgorithm' targeted for - // [Int]' types. - myAlgorithm(arrayOfInts, arrayOfInts.length) - -Advice: Put generic declarations in the same file where they are used ---------------------------------------------------------------------- - -The optimizer can only perform specializations if the definition of the generic -declaration is visible in the current Module. This can only occur if the -declaration is in the same file as the invocation of the generic. *NOTE* The -standard library is a special case. Definitions in the standard library are -visible in all modules and available for specialization. - -Advice: Allow the compiler to perform generic specialization ------------------------------------------------------------- - -The compiler can only specialize generic code if the call site and the callee -function are located in the same compilation unit. One trick that we can use to -allow compiler to optimize the callee function is to write code that performs a -type check in the same compilation unit as the callee function. The code behind -the type check then re-dispatches the call to the generic function - but this -time it has the type information. In the code sample below we've inserted a type -check into the function "play_a_game" and made the code run hundreds of times -faster. - -:: - - //Framework.swift: - - protocol Pingable { func ping() -> Self } - protocol Playable { func play() } - - extension Int : Pingable { - func ping() -> Int { return self + 1 } - } - - class Game : Playable { - var t : T - - init (_ v : T) {t = v} - - func play() { - for _ in 0...100_000_000 { t = t.ping() } - } - } - - func play_a_game(game : Playable ) { - // This check allows the optimizer to specialize the - // generic call 'play' - if let z = game as? Game { - z.play() - } else { - game.play() - } - } - - /// -------------- >8 - - // Application.swift: - - play_a_game(Game(10)) - - -The cost of large swift values -============================== - -In Swift, values keep a unique copy of their data. There are several advantages -to using value-types, like ensuring that values have independent state. When we -copy values (the effect of assignment, initialization, and argument passing) the -program will create a new copy of the value. For some large values these copies -could be time consuming and hurt the performance of the program. - -.. More on value types: -.. https://developer.apple.com/swift/blog/?id=10 - -Consider the example below that defines a tree using "value" nodes. The tree -nodes contain other nodes using a protocol. In computer graphics scenes are -often composed from different entities and transformations that can be -represented as values, so this example is somewhat realistic. - -.. See Protocol-Oriented-Programming: -.. https://developer.apple.com/videos/play/wwdc2015-408/ - -:: - - protocol P {} - struct Node : P { - var left, right : P? - } - - struct Tree { - var node : P? - init() { ... } - } - - -When a tree is copied (passed as an argument, initialized or assigned) the whole -tree needs to be copied. In the case of our tree this is an expensive operation -that requires many calls to malloc/free and a significant reference counting -overhead. - -However, we don't really care if the value is copied in memory as long as the -semantics of the value remains. - -Advice: Use copy-on-write semantics for large values ----------------------------------------------------- - -To eliminate the cost of copying large values adopt copy-on-write behavior. The -easiest way to implement copy-on-write is to compose existing copy-on-write data -structures, such as Array. Swift arrays are values, but the content of the array -is not copied around every time the array is passed as an argument because it -features copy-on-write traits. - -In our Tree example we eliminate the cost of copying the content of the tree by -wrapping it in an array. This simple change has a major impact on the -performance of our tree data structure, and the cost of passing the array as an -argument drops from being O(n), depending on the size of the tree to O(1). - -:: - - struct tree : P { - var node : [P?] - init() { - node = [ thing ] - } - } - - -There are two obvious disadvantages of using Array for COW semantics. The first -problem is that Array exposes methods like "append" and "count" that don't make -any sense in the context of a value wrapper. These methods can make the use of -the reference wrapper awkward. It is possible to work around this problem by -creating a wrapper struct that will hide the unused APIs and the optimizer will -remove this overhead, but this wrapper will not solve the second problem. The -Second problem is that Array has code for ensuring program safety and -interaction with Objective-C. Swift checks if indexed accesses fall within the -array bounds and when storing a value if the array storage needs to be extended. -These runtime checks can slow things down. - -An alternative to using Array is to implement a dedicated copy-on-write data -structure to replace Array as the value wrapper. The example below shows how to -construct such a data structure: - -.. Note: This solution is suboptimal for nested structs, and an addressor based -.. COW data structure would be more efficient. However at this point it's not -.. possible to implement addressors out of the standard library. - -.. More details in this blog post by Mike Ash: -.. https://www.mikeash.com/pyblog/friday-qa-2015-04-17-lets-build-swiftarray.html - -:: - - final class Ref { - var val : T - init(_ v : T) {val = v} - } - - struct Box { - var ref : Ref - init(_ x : T) { ref = Ref(x) } - - var value: T { - get { return ref.val } - set { - if (!isUniquelyReferencedNonObjC(&ref)) { - ref = Ref(newValue) - return - } - ref.val = newValue - } - } - } - -The type ``Box`` can replace the array in the code sample above. - -Unsafe code -=========== - -Swift classes are always reference counted. The swift compiler inserts code -that increments the reference count every time the object is accessed. -For example, consider the problem of scanning a linked list that's -implemented using classes. Scanning the list is done by moving a -reference from one node to the next: ``elem = elem.next``. Every time we move -the reference swift will increment the reference count of the ``next`` object -and decrement the reference count of the previous object. These reference -count operations are expensive and unavoidable when using Swift classes. - -:: - - final class Node { - var next: Node? - var data: Int - ... - } - - -Advice: Use unmanaged references to avoid reference counting overhead ---------------------------------------------------------------------- - -In performance-critical code you can use choose to use unmanaged -references. The ``Unmanaged`` structure allows developers to disable -automatic reference counting for a specific reference. - -:: - - var Ref : Unmanaged = Unmanaged.passUnretained(Head) - - while let Next = Ref.takeUnretainedValue().next { - ... - Ref = Unmanaged.passUnretained(Next) - } - - -Protocols -========= - -Advice: Mark protocols that are only satisfied by classes as class-protocols ----------------------------------------------------------------------------- - -Swift can limit protocols adoption to classes only. One advantage of marking -protocols as class-only is that the compiler can optimize the program based on -the knowledge that only classes satisfy a protocol. For example, the ARC memory -management system can easily retain (increase the reference count of an object) -if it knows that it is dealing with a class. Without this knowledge the compiler -has to assume that a struct may satisfy the protocol and it needs to be prepared -to retain or release non-trivial structures, which can be expensive. - -If it makes sense to limit the adoption of protocols to classes then mark -protocols as class-only protocols to get better runtime performance. - -:: - - protocol Pingable : class { func ping() -> Int } - -.. https://developer.apple.com/library/ios/documentation/Swift/Conceptual/Swift_Programming_Language/Protocols.html - - - -Footnotes -========= - -.. [#] A virtual method table or 'vtable' is a type specific table referenced by - instances that contains the addresses of the type's methods. Dynamic - dispatch proceeds by first looking up the table from the object and then - looking up the method in the table. - -.. [#] This is due to the compiler not knowing the exact function being called. - -.. [#] i.e. a direct load of a class's field or a direct call to a function. - -.. [#] Explain what COW is here. - -.. [#] In certain cases the optimizer is able to via inlining and ARC - optimization remove the retain, release causing no copy to occur. diff --git a/docs/Pattern Matching.md b/docs/Pattern Matching.md new file mode 100644 index 0000000000000..8c52d90ac29d2 --- /dev/null +++ b/docs/Pattern Matching.md @@ -0,0 +1,773 @@ +Pattern Matching +================ + +> **warning** +> +> This document was used in designing the pattern-matching features + +> of Swift 1.0. It has not been kept to date and does not describe the +> current or planned behavior of Swift. + +Elimination rules +----------------- + +When type theorists consider a programming language, we break it down +like this: + +- What are the kinds of fundamental and derived types in the language? +- For each type, what are its introduction rules, i.e. how do you get + values of that type? +- For each type, what are its elimination rules, i.e. how do you use + values of that type? + +Swift has a pretty small set of types right now: + +- Fundamental types: currently i1, i8, i16, i32, and i64; float and + double; eventually maybe others. +- Function types. +- Tuples. Heterogeneous fixed-length products. Swift's system provides + two basic kinds of element: positional and labelled. +- Arrays. Homogeneous fixed-length aggregates. +- Algebraic data types (ADTs), introduce by enum. Nominal closed + disjoint unions of heterogeneous types. +- Struct types. Nominal heterogeneous fixed-length products. +- Class types. Nominal, subtypeable heterogeneous fixed-length + products with identity. +- Protocol and protocol-composition types. + +In addition, each of the nominal types can be made generic; this doesn't +affect the overall introduction/elimination design because an +"unapplied" generic type isn't first-class (intentionally), and an +"applied" generic type behaves essentially like a non-generic type (also +intentionally). + +The point is that adding any other kind of type (e.g. SIMD vectors) +means that we need to consider its intro/elim rules. + +For most of these, intro rules are just a question of picking syntax, +and we don't really need a document for that. So let's talk elimination. +Generally, an elimination rule is a way at getting back to the +information the intro rule(s) wrote into the value. So what are the +specific elimination rules for these types? How do we use them, other +than in type-generic ways like passing them as arguments to calls? + +**Functions** are used by calling them. This is something of a special +case: some values of function type may carry data, there isn't really a +useful model for directly accessing it. Values of function type are +basically completely opaque, except that we do provide thin vs. thick +function types, which is potentially something we could pattern-match +on, although many things can introduce thunks and so the result would +not be reliable. + +**Scalars** are used by feeding them to primitive binary operators. This +is also something of a special case, because there's no useful way in +which scalars can be decomposed into separate values. + +**Tuples**, **structs**, and **classes** are used by projecting out +their elements. Classes may also be turned into an object of a supertype +(which is always a class). + +**Arrays** are used by projecting out slices and elements. + +**Existentials** are used by performing one of the operations that the +type is known to support. + +**ADTs** are used by projecting out elements of the current alternative, +but how we determine the current alternative? + +Alternatives for alternatives +----------------------------- + +I know of three basic designs for determining the current alternative of +an ADT: + +- Visitor pattern: there's some way of declaring a method on the full + ADT and then implementing it for each individual alternative. You do + this in OO languages mostly because there's no direct language + support for closed disjoint unions (as opposed to open disjoint + unions, which subclassing lets you achieve at some + performance cost). + - plus: doesn't require language support + - plus: easy to "overload" and provide different kinds of pattern + matching on the same type + - plus: straightforward to add interesting ADT-specific logic, + like matching a CallExpr instead of each of its N syntactic + forms + - plus: simple form of exhaustiveness checking + - minus: cases are separate functions, so data and control flow is + awkward + - minus: lots of boilerplate to enable + - minus: lots of boilerplate to use + - minus: nested pattern matching is awful +- Query functions: dynamic\_cast, dyn\_cast, isa, instanceof + - plus: easy to order and mix with other custom conditions + - plus: low syntactic overhead for testing the alternative if you + don't need to actually decompose + - minus: higher syntactic overhead for decomposition + - isa/instanceof pattern requires either a separate cast or + unsafe operations later + - dyn\_cast pattern needs a fresh variable declaration, which + is very awkward in complex conditions + - minus: exhaustiveness checking is basically out the window + - minus: some amount of boilerplate to enable +- Pattern matching + - plus: no boilerplate to enable + - plus: hugely reduced syntax to use if you want a full + decomposition + - plus: compiler-supported exhaustiveness checking + - plus: nested matching is natural + - plus: with pattern guards, natural mixing of custom conditions + - minus: syntactic overkill to just test for a specific + alternative (e.g. to filter it out) + - minus: needs boilerplate to project out a common member across + multiple/all alternatives + - minus: awkward to group alternatives (fallthrough is a simple + option but has issues) + - minus: traditionally completely autogenerated by compiler and + thus not very flexible + - minus: usually a new grammar production that's very ambiguous + with the expression grammar + - minus: somewhat fragile against adding extra data to an + alternative + +I feel that this strongly points towards using pattern matching as the +basic way of consuming ADTs, maybe with special dispensations for +querying the alternative and projecting out common members. + +Pattern matching was probably a foregone conclusion, but I wanted to +spell out that having ADTs in the language is what really forces our +hand because the alternatives are so bad. Once we need pattern-matching, +it makes sense to provide patterns for the other kinds of types as well. + +Selection statement +------------------- + +This is the main way we expect users to employ non-obvious pattern- +matching. We obviously need something with statement children, so this +has to be a statement. That's also fine because this kind of full +pattern match is very syntactically heavyweight, and nobody would want +to embed it in the middle of an expression. We also want a low-weight +matching expression, though, for relatively simple ADTs: + + stmt ::= stmt-switch + stmt-switch ::= 'switch' expr '{' switch-group+ '}' + switch-group ::= case-introducer+ stmt-brace-item+ + case-introducer ::= 'case' match-pattern-list case-guard? ':' + case-introducer ::= 'default' case-guard? ':' + case-guard ::= 'where' expr + match-pattern-list ::= match-pattern + match-pattern-list ::= match-pattern-list ',' match-pattern + +We can get away with using "switch" here because we're going to unify +both values and patterns under match-pattern. The works chiefly by +making decompositional binding a bit more awkward, but has the major +upside of reducing the likelihood of dumb mistakes (rebinding 'true', +for example), and it means that C-looking switches actually match our +semantics quite closely. The latter is something of a priority: a C +switch over an enum is actually pretty elegant --- well, except for all +the explicit scoping and 'break' statements, but the switching side of +it feels clean. + +### Default + +I keep going back and forth about having a "default" case-introducer. On +the one hand, I kind of want to encourage total matches. On the other +hand, (1) having it is consistent with C, (2) it's not an unnatural +style, and (3) there are cases where exhaustive switching isn't going to +be possible. We can certainly recommend complete matches in switches, +though. + +If we do have a 'default', I think it makes the most sense for it to be +semantically a complete match and therefore require it to be positioned +at the end (on pain of later matches being irrelevant). First, this +gives more sensible behavior to 'default where x.isPurple()', which +really doesn't seem like it should get reordered with the surrounding +cases; and second, it makes the matching story very straightforward. And +users who like to put 'default:' at the top won't accidentally get +unexpected behavior because coverage checking will immediately complain +about the fact that every case after an unguarded 'default' is obviously +dead. + +### Case groups + +A case-group lets you do the same thing for multiple cases without an +extra syntactic overhead (like a 'fallthrough' after every case). For +some types (e.g. classic functional linked lists) this is basically +pointless, but for a lot of other types (Int, enums, etc.) it's +pervasive. + +The most important semantic design point here is about bound variables +in a grouped case, e.g. (using 'var' as a "bind this variable" +introducer; see the pattern grammar): + + switch (pair) { + case (var x, 0): + case (0, var y): + return 1 + case (var x, var y) + return foo(x-1,y) + foo(x,y-1) + } + +It's tempting to just say that an unsound name binding (i.e. a name not +bound in all cases or bound to values of different types) is just always +an error, but I think that's probably not the way to go. There are two +things I have in mind here: first, these variables can be useful in +pattern guards even if they're not used in the case block itself, and +second, a well-chosen name can make a pattern much more +self-documenting. So I think it should only be an error to *refer* to an +unsound name binding. + +The most important syntactic design point here is whether to require (or +even allow) the 'case' keyword to be repeated for each case. In many +cases, it can be much more compact to allow a comma-separated list of +patterns after 'case': + + switch (day) { + case .Terrible, .Horrible, .NoGood, .VeryBad: + abort() + case .ActuallyPrettyReasonableWhenYouLookBackOnIt: + continue + } + +or even more so: + + case 0...2, 5...10, 14...18, 22...: + flagConditionallyAcceptableAge() + +On the other hand, if this list gets really long, the wrapping gets a +little weird: + + case .Terrible, .Horrible, .NoGood, .VeryBad, + .Awful, .Dreadful, .Appalling, .Horrendous, + .Deplorable, .Unpleasant, .Ghastly, .Dire: + abort() + +And while I think pattern guards should be able to apply to multiple +cases, it would be nice to allow different cases in a group to have +different pattern guards: + + case .None: + case .Some(var c) where c.isSpace() || c.isASCIIControl(): + skipToEOL() + +So really I think we should permit multiple 'case' introducers: + + case .Terrible, .Horrible, .NoGood, .VeryBad: + case .Awful, .Dreadful, .Appalling, .Horrendous: + case .Deplorable, .Unpleasant, .Ghastly, .Dire: + abort() + +With the rule that a pattern guard can only use bindings that are sound +across its guarded patterns (those within the same 'case'), and the +statement itself can only use bindings that are sound across all of the +cases. A reference that refers to an unsound binding is an error; lookup +doesn't just ignore the binding. + +### Scoping + +Despite the lack of grouping braces, the semantics are that the +statements in each case-group form their own scope, and falling off the +end causes control to resume at the end of the switch statement — i.e. +"implicit break", not "implicit fallthrough". + +Chris seems motivated to eventually add an explicit 'fallthrough' +statement. If we did this, my preference would be to generalize it by +allowing the match to be performed again with a new value, e.g. +`fallthrough(something)`{.sourceCode}, at least optionally. I think +having local functions removes a lot of the impetus, but not so much as +to render the feature worthless. + +Syntactically, braces and the choice of case keywords are all bound +together. The thinking goes as follows. In Swift, statement scopes are +always grouped by braces. It's natural to group the cases with braces as +well. Doing both lets us avoid a 'case' keyword, but otherwise it leads +to ugly style, because either the last case ends in two braces on the +same line or cases have to further indented. Okay, it's easy enough to +not require braces on the match, with the grammar saying that cases are +just greedily consumed — there's no ambiguity here because the switch +statement is necessarily within braces. But that leaves the code without +a definitive end to the cases, and the closing braces end up causing a +lot of unnecessary vertical whitespace, like so: + + switch (x) + case .foo { + … + } + case .bar { + … + } + +So instead, let's require the switch statement to have braces, and we'll +allow the cases to be written without them: + + switch (x) { + case .foo: + … + case .bar: + … + } + +That's really a lot prettier, except it breaks the rule about always +grouping scopes with braces (we *definitely* want different cases to +establish different scopes). Something has to give, though. + +We require the trailing colon because it's a huge cue for separating +things, really making single-line cases visually appealing, and the fact +that it doesn't suggest closing punctuation is a huge boon. It's also +directly precedented in C, and it's even roughly the right grammatical +function. + +### Case selection semantics + +The semantics of a switch statement are to first evaluate the value +operand, then proceed down the list of case-introducers and execute the +statements for the switch-group that had the first satisfied introducer. + +It is an error if a case-pattern can never trigger because earlier cases +are exhaustive. Some kinds of pattern (like 'default' cases and '\_') +are obviously exhaustive by themselves, but other patterns (like +patterns on properties) can be much harder to reason about +exhaustiveness for, and of course pattern guards can make this outright +undecidable. It may be easiest to apply very straightforward rules (like +"ignore guarded patterns") for the purposes of deciding whether the +program is actually ill-formed; anything else that we can prove is +unreachable would only merit a warning. We'll probably also want a way +to say explicitly that a case can never occur (with semantics like +llvm\_unreachable, i.e. a reliable runtime failure unless that kind of +runtime safety checking is disabled at compile-time). + +A 'default' is satisfied if it has no guard or if the guard evaluates to +true. + +A 'case' is satisfied if the pattern is satisfied and, if there's a +guard, the guard evaluates to true after binding variables. The guard is +not evaluated if the pattern is not fully satisfied. We'll talk about +satisfying a pattern later. + +### Non-exhaustive switches + +Since falling out of a statement is reasonable behavior in an imperative +language — in contrast to, say, a functional language where you're in an +expression and you need to produce a value — there's a colorable +argument that non-exhaustive matches should be okay. I dislike this, +however, and propose that it should be an error to make an +non-exhaustive switch; people who want non-exhaustive matches can +explicitly put in default cases. Exhaustiveness actually isn't that +difficult to check, at least over ADTs. It's also really the behavior +that I would expect from the syntax, or at least implicitly falling out +seems dangerous in a way that nonexhaustive checking doesn't. The +complications with checking exhaustiveness are pattern guards and +matching expressions. The obvious conservatively-safe rule is to say +"ignore cases with pattern guards or matching expressions during +exhaustiveness checking", but some people really want to write "where x +< 10" and "where x >= 10", and I can see their point. At the same +time, we really don't want to go down that road. + +Other uses of patterns +---------------------- + +Patterns come up (or could potentially come up) in a few other places in +the grammar: + +### Var bindings + +Variable bindings only have a single pattern, which has to be +exhaustive, which also means there's no point in supporting guards here. +I think we just get this: + + decl-var ::= 'var' attribute-list? pattern-exhaustive value-specifier + +### Function parameters + +The functional languages all permit you to directly pattern-match in the +function declaration, like this example from SML: + + fun length nil = 0 + | length (a::b) = 1 + length b + +This is really convenient, but there's probably no reasonable analogue +in Swift. One specific reason: we want functions to be callable with +keyword arguments, but if you don't give all the parameters their own +names, that won't work. + +The current Swift approximation is: + + func length(list : List) : Int { + switch list { + case .nil: return 0 + case .cons(_,var tail): return 1 + length(tail) + } + } + +That's quite a bit more syntax, but it's mostly the extra braces from +the function body. We could remove those with something like this: + + func length(list : List) : Int = switch list { + case .nil: return 0 + case .cons(_,var tail): return 1 + length(tail) + } + +Anyway, that's easy to add later if we see the need. + +### Assignment + +This is a bit iffy. It's a lot like var bindings, but it doesn't have a +keyword, so it's really kind of ambiguous given the pattern grammar. + +Also, l-value patterns are weird. I can come up with semantics for this, +but I don't know what the neighbors will think: + + var perimeter : double + .feet(x) += yard.dimensions.height // returns Feet, which has one constructor, :feet. + .feet(x) += yard.dimensions.width + +It's probably better to just have l-value tuple expressions and not try +to work in arbitrary patterns. + +### Pattern-match expression + +This is an attempt to provide that dispensation for query functions we +were talking about. + +I think this should bind looser than any binary operators except +assignments; effectively we should have: + + expr-binary ::= # most of the current expr grammar + + expr ::= expr-binary + expr ::= expr-binary 'is' expr-primary pattern-guard? + +The semantics are that this evaluates to true if the pattern and +pattern-guard are satisfied. + +#### 'is' or 'isa' + +Perl and Ruby use '=\~' as the regexp pattern-matching operator, which +is both obscure and really looks like an assignment operator, so I'm +stealing Joe's 'is' operator, which is currently used for dynamic +type-checks. I'm of two minds about this: I like 'is' a lot for +value-matching, but not for dynamic type-checks. + +One possibility would be to use 'is' as the generic pattern-matching +operator but use a different spelling (like 'isa') for dynamic +type-checks, including the 'is' pattern. This would give us "x isa +NSObject" as an expression and "case isa NSObject:" as a case selector, +both of which I feel read much better. But in this proposal, we just use +a single operator. + +Other alternatives to 'is' include 'matches' (reads very naturally but +is somewhat verbose) or some sort of novel operator like '\~\~'. + +Note that this impacts a discussion in the section below about +expression patterns. + +#### Dominance + +I think that this feature is far more powerful if the name bindings, +type-refinements, etc. from patterns are available in code for which a +trivial analysis would reveal that the result of the expression is true. +For example: + + if s is Window where x.isVisible { + // can use Window methods on x here + } + +Taken a bit further, we can remove the need for 'where' in the +expression form: + + if x is Window && x.isVisible { ... } + +That might be problematic without hard-coding the common control-flow +operators, though. (As well as hardcoding some assumptions about +Bool.convertToLogicValue...) + +Pattern grammar +--------------- + +The usual syntax rule from functional languages is that the pattern +grammar mirrors the introduction-rule expression grammar, but parses a +pattern wherever you would otherwise put an expression. This means that, +for example, if we add array literal expressions, we should also add a +corresponding array literal pattern. I think that principle is very +natural and worth sticking to wherever possible. + +### Two kinds of pattern + +We're blurring the distinction between patterns and expressions a lot +here. My current thinking is that this simplifies things for the +programmer --- the user concept becomes basically "check whether we're +equal to this expression, but allow some holes and some more complex +'matcher' values". But it's possible that it instead might be really +badly confusing. We'll see! It'll be fun! + +This kind of forces us to have parallel pattern grammars for the two +major clients: + +- Match patterns are used in `switch`{.sourceCode} and + `matches`{.sourceCode}, where we're decomposing something with a + real possibility of failing. This means that expressions are okay in + leaf positions, but that name-bindings need to be explicitly + advertised in some way to reasonably disambiguate them + from expressions. +- Exhaustive patterns are used in `var`{.sourceCode} declarations and + function signatures. They're not allowed to be non-exhaustive, so + having a match expression doesn't make any sense. Name bindings are + common and so shouldn't be penalized. + +You might think that having a "pattern" as basic as `foo`{.sourceCode} +mean something different in two different contexts would be confusing, +but actually I don't think people will generally think of these as the +same production — you might if you were in a functional language where +you really can decompose in a function signature, but we don't allow +that, and I think that will serve to divide them in programmers' minds. +So we can get away with some things. :) + +### Binding patterns + +In general, a lot of these productions are the same, so I'm going to +talk about `*`-patterns, with some specific special rules that only +apply to specific pattern kinds. + + *-pattern ::= '_' + +A single-underscore identifier is always an "ignore" pattern. It matches +anything, but does not bind it to a variable. + + exhaustive-pattern ::= identifier + match-pattern ::= '?' identifier + +Any more complicated identifier is a variable-binding pattern. It is +illegal to bind the same identifier multiple times within a pattern. +However, the variable does come into scope immediately, so in a match +pattern you can have a latter expression which refers to an +already-bound variable. I'm comfortable with constraining this to only +work "conveniently" left-to-right and requiring more complicated matches +to use guard expressions. + +In a match pattern, variable bindings must be prefixed with a ? to +disambiguate them from an expression consisting of a variable reference. +I considered using 'var' instead, but using punctuation means we don't +need a space, which means this is much more compact in practice. + +### Annotation patterns + + exhaustive-pattern ::= exhaustive-pattern ':' type + +In an exhaustive pattern, you can annotate an arbitrary sub-pattern with +a type. This is useful in an exhaustive pattern: the type of a variable +isn't always inferable (or correctly inferable), and types in function +signatures are generally outright required. It's not as useful in a +match pattern, and the colon can be grammatically awkward there, so we +disallow it. + +### 'is' patterns + + match-pattern ::= 'is' type + +This pattern is satisfied if the dynamic type of the matched value +"satisfies" the named type: + +> - if the named type is an Objective-C class type, the dynamic type +> must be a class type, and an 'isKindOf:' check is performed; +> - if the named type is a Swift class type, the dynamic type must be +> a class type, and a subtype check is performed; +> - if the named type is a metatype, the dynamic type must be a +> metatype, and the object type of the dynamic type must satisfy the +> object type of the named type; +> - otherwise the named type must equal the dynamic type. + +This inquiry is about dynamic types; archetypes and existentials are +looked through. + +The pattern is ill-formed if it provably cannot be satisfied. + +In a 'switch' statement, this would typically appear like this: + + case is NSObject: + +It can, however, appear in recursive positions: + + case (is NSObject, is NSObject): + +#### Ambiguity with type value matching + +There is a potential point of confusion here with dynamic type checking +(done by an 'is' pattern) vs. value equality on type objects (done by an +expression pattern where the expression is of metatype type. This is +resolved by the proposal (currently outstanding but generally accepted, +I think) to disallow naked references to type constants and instead +require them to be somehow decorated. + +That is, this pattern requires the user to write something like this: + + case is NSObject: + +It is quite likely that users will often accidentally write something +like this: + + case NSObject: + +It would be very bad if that were actually accepted as a valid +expression but with the very different semantics of testing equality of +type objects. For the most part, type-checking would reject that as +invalid, but a switch on (say) a value of archetype type would generally +work around that. + +However, we have an outstanding proposal to generally forbid 'NSObject' +from appearing as a general expression; the user would have to decorate +it like the following, which would let us eliminate the common mistake: + + case NSObject.type: + +#### Type refinement + +If the value matched is immediately the value of a local variable, I +think it would be really useful if this pattern could introduce a type +refinement within its case, so that the local variable would have the +refined type within that scope. However, making this kind of type +refinement sound would require us to prevent there from being any sort +of mutable alias of the local variable under an unrefined type. That's +usually going to be fine in Swift because we usually don't permit the +address of a local to escape in a way that crosses statement boundaries. +However, closures are a major problem for this model. If we had +immutable local bindings --- and, better yet, if they were the default +--- this problem would largely go away. + +This sort of type refinement could also be a problem with code like: + + while expr is ParenExpr { + expr = expr.getSubExpr() + } + +It's tricky. + +### "Call" patterns + + match-pattern ::= match-pattern-identifier match-pattern-tuple? + match-pattern-identifier ::= '.' identifier + match-pattern-identifier ::= match-pattern-identifier-tower + match-pattern-identifier-tower ::= identifier + match-pattern-identifier-tower ::= identifier + match-pattern-identifier-tower ::= match-pattern-identifier-tower '.' identifier + +A match pattern can resemble a global name or a call to a global name. +The global name is resolved as normal, and then the pattern is +interpreted according to what is found: + +- If the name resolves to a type, then the dynamic type of the matched + value must match the named type (according to the rules below for + 'is' patterns). It is okay for this to be trivially true. + + In addition, there must be an non-empty arguments clause, and each + element in the clause must have an identifier. For each element, the + identifier must correspond to a known property of the named type, + and the value of that property must satisfy the element pattern. + +- If the name resolves to a enum element, then the dynamic type of the + matched value must match the enum type as discussed above, and the + value must be of the specified element. There must be an arguments + clause if and only if the element has a value type. If so, the value + of the element is matched against the clause pattern. +- Otherwise, the argument clause (if present) must also be + syntactically valid as an expression, and the entire pattern is + reinterpreted as an expression. + +This is all a bit lookup-sensitive, which makes me uncomfortable, but +otherwise I think it makes for attractive syntax. I'm also a little +worried about the way that, say, `f(x)`{.sourceCode} is always an +expression but `A(x)`{.sourceCode} is a pattern. Requiring property +names when matching properties goes some way towards making that okay. + +I'm not totally sold on not allowing positional matching against struct +elements; that seems unfortunate in cases where positionality is +conventionally unambiguous, like with a point. + +Matching against struct types requires arguments because this is +intended to be used for structure decomposition, not dynamic type +testing. For the latter, an 'is' pattern should be used. + +### Expression patterns + + match-pattern ::= expression + +When ambiguous, match patterns are interpreted using a pattern-specific +production. I believe it should be true that, in general, match patterns +for a production accept a strict superset of valid expressions, so that +(e.g.) we do not need to disambiguate whether an open paren starts a +tuple expression or a tuple pattern, but can instead just aggressively +parse as a pattern. Note that binary operators can mean that, using this +strategy, we sometimes have to retroactively rewrite a pattern as an +expression. + +It's always possible to disambiguate something as an expression by doing +something not allowing in patterns, like using a unary operator or +calling an identity function; those seem like unfortunate language +solutions, though. + +### Satisfying an expression pattern + +A value satisfies an expression pattern if the match operation succeeds. +I think it would be natural for this match operation to be spelled the +same way as that match-expression operator, so e.g. a member function +called 'matches' or a global binary operator called '\~' or whatever. + +The lookup of this operation poses some interesting questions. In +general, the operation itself is likely to be associated with the +intended type of the expression pattern, but that type will often +require refinement from the type of the matched value. + +For example, consider a pattern like this: + + case 0...10: + +We should be able to use this pattern when switching on a value which is +not an Int, but if we type-check the expression on its own, we will +assign it the type Range<Int>, which will not necessarily permit +us to match (say) a UInt8. + +### Order of evaluation of patterns + +I'd like to keep the order of evaluation and testing of expressions +within a pattern unspecified if I can; I imagine that there should be a +lot of cases where we can rule out a case using a cheap test instead of +a more expensive one, and it would suck to have to run the expensive one +just to have cleaner formal semantics. Specifically, I'm worried about +cases like `case [foo(), 0]:`{.sourceCode}; if we can test against 0 +before calling `foo()`{.sourceCode}, that would be great. Also, if a +name is bound and then used directly as an expression later on, it would +be nice to have some flexibility about which value is actually copied +into the variable, but this is less critical. + + *-pattern ::= *-pattern-tuple + *-pattern-tuple ::= '(' *-pattern-tuple-element-list? '...'? ')' + *-pattern-tuple-element-list ::= *-pattern-tuple-element + *-pattern-tuple-element-list ::= *-pattern-tuple-element ',' pattern-tuple-element-list + *-pattern-tuple-element ::= *-pattern + *-pattern-tuple-element ::= identifier '=' *-pattern + +Tuples are interesting because of the labelled / non-labelled +distinction. Especially with labelled elements, it is really nice to be +able to ignore all the elements you don't care about. This grammar +permits some prefix or set of labels to be matched and the rest to be +ignored. + +Miscellaneous +------------- + +It would be interesting to allow overloading / customization of +pattern-matching. We may find ourselves needing to do something like +this to support non-fragile pattern matching anyway (if there's some set +of restrictions that make it reasonable to permit that). The obvious +idea of compiling into the visitor pattern is a bit compelling, although +control flow would be tricky — we'd probably need the generated code to +throw an exception. Alternatively, we could let the non-fragile type +convert itself into a fragile type for purposes of pattern matching. + +If we ever allow infix ADT constructors, we'll need to allow them in +patterns as well. + +Eventually, we will build regular expressions into the language, and we +will allow them directly as patterns and even bind grouping expressions +into user variables. + +John. diff --git a/docs/Pattern Matching.rst b/docs/Pattern Matching.rst deleted file mode 100644 index d0bfedd6c2dfd..0000000000000 --- a/docs/Pattern Matching.rst +++ /dev/null @@ -1,815 +0,0 @@ -.. @raise litre.TestsAreMissing -.. _PatternMatching: - -Pattern Matching -================ - -.. warning:: This document was used in designing the pattern-matching features - of Swift 1.0. It has not been kept to date and does not describe the current - or planned behavior of Swift. - -Elimination rules ------------------ - -When type theorists consider a programming language, we break it down like this: - -* What are the kinds of fundamental and derived types in the language? -* For each type, what are its introduction rules, i.e. how do you get - values of that type? -* For each type, what are its elimination rules, i.e. how do you use - values of that type? - -Swift has a pretty small set of types right now: - -* Fundamental types: currently i1, i8, i16, i32, and i64; - float and double; eventually maybe others. -* Function types. -* Tuples. Heterogeneous fixed-length products. Swift's system - provides two basic kinds of element: positional and labelled. -* Arrays. Homogeneous fixed-length aggregates. -* Algebraic data types (ADTs), introduce by enum. Nominal closed - disjoint unions of heterogeneous types. -* Struct types. Nominal heterogeneous fixed-length products. -* Class types. Nominal, subtypeable heterogeneous fixed-length products - with identity. -* Protocol and protocol-composition types. - -In addition, each of the nominal types can be made generic; this -doesn't affect the overall introduction/elimination design because an -"unapplied" generic type isn't first-class (intentionally), and an -"applied" generic type behaves essentially like a non-generic type -(also intentionally). - -The point is that adding any other kind of type (e.g. SIMD vectors) -means that we need to consider its intro/elim rules. - -For most of these, intro rules are just a question of picking syntax, and we -don't really need a document for that. So let's talk elimination. Generally, an -elimination rule is a way at getting back to the information the intro rule(s) -wrote into the value. So what are the specific elimination rules for these -types? How do we use them, other than in type-generic ways like passing them as -arguments to calls? - -**Functions** are used by calling them. This is something of a special case: -some values of function type may carry data, there isn't really a useful model -for directly accessing it. Values of function type are basically completely -opaque, except that we do provide thin vs. thick function types, which is -potentially something we could pattern-match on, although many things can -introduce thunks and so the result would not be reliable. - -**Scalars** are used by feeding them to primitive binary operators. This is -also something of a special case, because there's no useful way in which scalars -can be decomposed into separate values. - -**Tuples**, **structs**, and **classes** are used by projecting out -their elements. Classes may also be turned into an object of a -supertype (which is always a class). - -**Arrays** are used by projecting out slices and elements. - -**Existentials** are used by performing one of the operations that the -type is known to support. - -**ADTs** are used by projecting out elements of the current alternative, but how -we determine the current alternative? - -Alternatives for alternatives ------------------------------ - -I know of three basic designs for determining the current alternative of an ADT: - -* Visitor pattern: there's some way of declaring a method on the full ADT and - then implementing it for each individual alternative. You do this in OO - languages mostly because there's no direct language support for closed - disjoint unions (as opposed to open disjoint unions, which subclassing lets - you achieve at some performance cost). - - * plus: doesn't require language support - * plus: easy to "overload" and provide different kinds of pattern matching on - the same type - * plus: straightforward to add interesting ADT-specific logic, like matching a - CallExpr instead of each of its N syntactic forms - * plus: simple form of exhaustiveness checking - * minus: cases are separate functions, so data and control flow is awkward - * minus: lots of boilerplate to enable - * minus: lots of boilerplate to use - * minus: nested pattern matching is awful - -* Query functions: dynamic_cast, dyn_cast, isa, instanceof - - * plus: easy to order and mix with other custom conditions - * plus: low syntactic overhead for testing the alternative if you don't need - to actually decompose - * minus: higher syntactic overhead for decomposition - - * isa/instanceof pattern requires either a separate cast or unsafe - operations later - * dyn_cast pattern needs a fresh variable declaration, which is very awkward - in complex conditions - - * minus: exhaustiveness checking is basically out the window - * minus: some amount of boilerplate to enable - -* Pattern matching - - * plus: no boilerplate to enable - * plus: hugely reduced syntax to use if you want a full decomposition - * plus: compiler-supported exhaustiveness checking - * plus: nested matching is natural - * plus: with pattern guards, natural mixing of custom conditions - * minus: syntactic overkill to just test for a specific alternative - (e.g. to filter it out) - * minus: needs boilerplate to project out a common member across - multiple/all alternatives - * minus: awkward to group alternatives (fallthrough is a simple option - but has issues) - * minus: traditionally completely autogenerated by compiler and thus - not very flexible - * minus: usually a new grammar production that's very ambiguous with - the expression grammar - * minus: somewhat fragile against adding extra data to an alternative - -I feel that this strongly points towards using pattern matching as the basic way -of consuming ADTs, maybe with special dispensations for querying the alternative -and projecting out common members. - -Pattern matching was probably a foregone conclusion, but I wanted to spell out -that having ADTs in the language is what really forces our hand because the -alternatives are so bad. Once we need pattern-matching, it makes sense to -provide patterns for the other kinds of types as well. - -Selection statement -------------------- - -This is the main way we expect users to employ non-obvious pattern- matching. We -obviously need something with statement children, so this has to be a -statement. That's also fine because this kind of full pattern match is very -syntactically heavyweight, and nobody would want to embed it in the middle of an -expression. We also want a low-weight matching expression, though, for -relatively simple ADTs:: - - stmt ::= stmt-switch - stmt-switch ::= 'switch' expr '{' switch-group+ '}' - switch-group ::= case-introducer+ stmt-brace-item+ - case-introducer ::= 'case' match-pattern-list case-guard? ':' - case-introducer ::= 'default' case-guard? ':' - case-guard ::= 'where' expr - match-pattern-list ::= match-pattern - match-pattern-list ::= match-pattern-list ',' match-pattern - -We can get away with using "switch" here because we're going to unify -both values and patterns under match-pattern. The works chiefly by -making decompositional binding a bit more awkward, but has the major -upside of reducing the likelihood of dumb mistakes (rebinding 'true', -for example), and it means that C-looking switches actually match our -semantics quite closely. The latter is something of a priority: a C -switch over an enum is actually pretty elegant --- well, except for -all the explicit scoping and 'break' statements, but the switching -side of it feels clean. - -Default -....... - -I keep going back and forth about having a "default" case-introducer. -On the one hand, I kind of want to encourage total matches. On the -other hand, (1) having it is consistent with C, (2) it's not an -unnatural style, and (3) there are cases where exhaustive switching -isn't going to be possible. We can certainly recommend complete -matches in switches, though. - -If we do have a 'default', I think it makes the most sense for it to be -semantically a complete match and therefore require it to be -positioned at the end (on pain of later matches being irrelevant). -First, this gives more sensible behavior to 'default where -x.isPurple()', which really doesn't seem like it should get reordered -with the surrounding cases; and second, it makes the matching story -very straightforward. And users who like to put 'default:' at the top -won't accidentally get unexpected behavior because coverage checking -will immediately complain about the fact that every case after an -unguarded 'default' is obviously dead. - -Case groups -........... - -A case-group lets you do the same thing for multiple cases without an -extra syntactic overhead (like a 'fallthrough' after every case). For -some types (e.g. classic functional linked lists) this is basically -pointless, but for a lot of other types (Int, enums, etc.) it's -pervasive. - -The most important semantic design point here is about bound variables -in a grouped case, e.g. (using 'var' as a "bind this variable" introducer; -see the pattern grammar):: - - switch (pair) { - case (var x, 0): - case (0, var y): - return 1 - case (var x, var y) - return foo(x-1,y) + foo(x,y-1) - } - -It's tempting to just say that an unsound name binding (i.e. a name -not bound in all cases or bound to values of different types) is just -always an error, but I think that's probably not the way to go. There -are two things I have in mind here: first, these variables can be -useful in pattern guards even if they're not used in the case block -itself, and second, a well-chosen name can make a pattern much more -self-documenting. So I think it should only be an error to *refer* to -an unsound name binding. - -The most important syntactic design point here is whether to require -(or even allow) the 'case' keyword to be repeated for each case. In -many cases, it can be much more compact to allow a comma-separated -list of patterns after 'case':: - - switch (day) { - case .Terrible, .Horrible, .NoGood, .VeryBad: - abort() - case .ActuallyPrettyReasonableWhenYouLookBackOnIt: - continue - } - -or even more so:: - - case 0...2, 5...10, 14...18, 22...: - flagConditionallyAcceptableAge() - -On the other hand, if this list gets really long, the wrapping gets a -little weird:: - - case .Terrible, .Horrible, .NoGood, .VeryBad, - .Awful, .Dreadful, .Appalling, .Horrendous, - .Deplorable, .Unpleasant, .Ghastly, .Dire: - abort() - -And while I think pattern guards should be able to apply to multiple -cases, it would be nice to allow different cases in a group to have -different pattern guards:: - - case .None: - case .Some(var c) where c.isSpace() || c.isASCIIControl(): - skipToEOL() - -So really I think we should permit multiple 'case' introducers:: - - case .Terrible, .Horrible, .NoGood, .VeryBad: - case .Awful, .Dreadful, .Appalling, .Horrendous: - case .Deplorable, .Unpleasant, .Ghastly, .Dire: - abort() - -With the rule that a pattern guard can only use bindings that are -sound across its guarded patterns (those within the same 'case'), and -the statement itself can only use bindings that are sound across all -of the cases. A reference that refers to an unsound binding is an -error; lookup doesn't just ignore the binding. - -Scoping -....... - -Despite the lack of grouping braces, the semantics are that the statements in -each case-group form their own scope, and falling off the end causes control to -resume at the end of the switch statement — i.e. "implicit break", not "implicit -fallthrough". - -Chris seems motivated to eventually add an explicit 'fallthrough' -statement. If we did this, my preference would be to generalize it by -allowing the match to be performed again with a new value, e.g. -:code:`fallthrough(something)`, at least optionally. I think having -local functions removes a lot of the impetus, but not so much as to -render the feature worthless. - -Syntactically, braces and the choice of case keywords are all bound -together. The thinking goes as follows. In Swift, statement scopes are always -grouped by braces. It's natural to group the cases with braces as well. Doing -both lets us avoid a 'case' keyword, but otherwise it leads to ugly style, -because either the last case ends in two braces on the same line or cases have -to further indented. Okay, it's easy enough to not require braces on the match, -with the grammar saying that cases are just greedily consumed — there's no -ambiguity here because the switch statement is necessarily within braces. But -that leaves the code without a definitive end to the cases, and the closing -braces end up causing a lot of unnecessary vertical whitespace, like so:: - - switch (x) - case .foo { - … - } - case .bar { - … - } - -So instead, let's require the switch statement to have braces, and -we'll allow the cases to be written without them:: - - switch (x) { - case .foo: - … - case .bar: - … - } - -That's really a lot prettier, except it breaks the rule about always grouping -scopes with braces (we *definitely* want different cases to establish different -scopes). Something has to give, though. - -We require the trailing colon because it's a huge cue for separating -things, really making single-line cases visually appealing, and the -fact that it doesn't suggest closing punctuation is a huge boon. It's -also directly precedented in C, and it's even roughly the right -grammatical function. - -Case selection semantics -........................ - -The semantics of a switch statement are to first evaluate the value -operand, then proceed down the list of case-introducers and execute -the statements for the switch-group that had the first satisfied -introducer. - -It is an error if a case-pattern can never trigger because earlier -cases are exhaustive. Some kinds of pattern (like 'default' cases -and '_') are obviously exhaustive by themselves, but other patterns -(like patterns on properties) can be much harder to reason about -exhaustiveness for, and of course pattern guards can make this -outright undecidable. It may be easiest to apply very straightforward -rules (like "ignore guarded patterns") for the purposes of deciding -whether the program is actually ill-formed; anything else that we can -prove is unreachable would only merit a warning. We'll probably -also want a way to say explicitly that a case can never occur (with -semantics like llvm_unreachable, i.e. a reliable runtime failure unless -that kind of runtime safety checking is disabled at compile-time). - -A 'default' is satisfied if it has no guard or if the guard evaluates to true. - -A 'case' is satisfied if the pattern is satisfied and, if there's a guard, -the guard evaluates to true after binding variables. The guard is not -evaluated if the pattern is not fully satisfied. We'll talk about satisfying -a pattern later. - -Non-exhaustive switches -....................... - -Since falling out of a statement is reasonable behavior in an -imperative language — in contrast to, say, a functional language where -you're in an expression and you need to produce a value — there's a -colorable argument that non-exhaustive matches should be okay. I -dislike this, however, and propose that it should be an error to -make an non-exhaustive switch; people who want non-exhaustive matches -can explicitly put in default cases. -Exhaustiveness actually isn't that difficult to check, at least over -ADTs. It's also really the behavior that I would expect from the -syntax, or at least implicitly falling out seems dangerous in a way -that nonexhaustive checking doesn't. The complications with checking -exhaustiveness are pattern guards and matching expressions. The -obvious conservatively-safe rule is to say "ignore cases with pattern -guards or matching expressions during exhaustiveness checking", but -some people really want to write "where x < 10" and "where x >= 10", -and I can see their point. At the same time, we really don't want to -go down that road. - -Other uses of patterns ----------------------- - -Patterns come up (or could potentially come up) in a few other places -in the grammar: - -Var bindings -............ - -Variable bindings only have a single pattern, which has to be exhaustive, which -also means there's no point in supporting guards here. I think we just get -this:: - - decl-var ::= 'var' attribute-list? pattern-exhaustive value-specifier - -Function parameters -................... - -The functional languages all permit you to directly pattern-match in the -function declaration, like this example from SML:: - - fun length nil = 0 - | length (a::b) = 1 + length b - -This is really convenient, but there's probably no reasonable analogue in -Swift. One specific reason: we want functions to be callable with keyword -arguments, but if you don't give all the parameters their own names, that won't -work. - -The current Swift approximation is:: - - func length(list : List) : Int { - switch list { - case .nil: return 0 - case .cons(_,var tail): return 1 + length(tail) - } - } - -That's quite a bit more syntax, but it's mostly the extra braces from the -function body. We could remove those with something like this:: - - func length(list : List) : Int = switch list { - case .nil: return 0 - case .cons(_,var tail): return 1 + length(tail) - } - -Anyway, that's easy to add later if we see the need. - -Assignment -.......... - -This is a bit iffy. It's a lot like var bindings, but it doesn't have a keyword, -so it's really kind of ambiguous given the pattern grammar. - -Also, l-value patterns are weird. I can come up with semantics for this, but I -don't know what the neighbors will think:: - - var perimeter : double - .feet(x) += yard.dimensions.height // returns Feet, which has one constructor, :feet. - .feet(x) += yard.dimensions.width - -It's probably better to just have l-value tuple expressions and not -try to work in arbitrary patterns. - -Pattern-match expression -........................ - -This is an attempt to provide that dispensation for query functions we were -talking about. - -I think this should bind looser than any binary operators except assignments; -effectively we should have:: - - expr-binary ::= # most of the current expr grammar - - expr ::= expr-binary - expr ::= expr-binary 'is' expr-primary pattern-guard? - -The semantics are that this evaluates to true if the pattern and -pattern-guard are satisfied. - -'is' or 'isa' -````````````` - -Perl and Ruby use '=~' as the regexp pattern-matching operator, which -is both obscure and really looks like an assignment operator, so I'm -stealing Joe's 'is' operator, which is currently used for dynamic -type-checks. I'm of two minds about this: I like 'is' a lot for -value-matching, but not for dynamic type-checks. - -One possibility would be to use 'is' as the generic pattern-matching -operator but use a different spelling (like 'isa') for dynamic -type-checks, including the 'is' pattern. This would give us -"x isa NSObject" as an expression and "case isa NSObject:" as a -case selector, both of which I feel read much better. But in this -proposal, we just use a single operator. - -Other alternatives to 'is' include 'matches' (reads very naturally but -is somewhat verbose) or some sort of novel operator like '~~'. - -Note that this impacts a discussion in the section below about -expression patterns. - -Dominance -````````` - -I think that this feature is far more powerful if the name bindings, -type-refinements, etc. from patterns are available in code for which a -trivial analysis would reveal that the result of the expression is -true. For example:: - - if s is Window where x.isVisible { - // can use Window methods on x here - } - -Taken a bit further, we can remove the need for 'where' in the -expression form:: - - if x is Window && x.isVisible { ... } - -That might be problematic without hard-coding the common -control-flow operators, though. (As well as hardcoding some -assumptions about Bool.convertToLogicValue...) - -Pattern grammar ---------------- - -The usual syntax rule from functional languages is that the pattern -grammar mirrors the introduction-rule expression grammar, but parses a -pattern wherever you would otherwise put an expression. This means -that, for example, if we add array literal expressions, we should also -add a corresponding array literal pattern. I think that principle is -very natural and worth sticking to wherever possible. - -Two kinds of pattern -.................... - -We're blurring the distinction between patterns and expressions a lot -here. My current thinking is that this simplifies things for the -programmer --- the user concept becomes basically "check whether we're -equal to this expression, but allow some holes and some more complex -'matcher' values". But it's possible that it instead might be really -badly confusing. We'll see! It'll be fun! - -This kind of forces us to have parallel pattern grammars for the two -major clients: - -- Match patterns are used in :code:`switch` and :code:`matches`, where - we're decomposing something with a real possibility of failing. - This means that expressions are okay in leaf positions, but that - name-bindings need to be explicitly advertised in some way to - reasonably disambiguate them from expressions. -- Exhaustive patterns are used in :code:`var` declarations - and function signatures. They're not allowed to be non-exhaustive, - so having a match expression doesn't make any sense. Name bindings - are common and so shouldn't be penalized. - -You might think that having a "pattern" as basic as :code:`foo` mean -something different in two different contexts would be confusing, but -actually I don't think people will generally think of these as the -same production — you might if you were in a functional language where -you really can decompose in a function signature, but we don't allow -that, and I think that will serve to divide them in programmers' minds. -So we can get away with some things. :) - -Binding patterns -................ - -In general, a lot of these productions are the same, so I'm going to -talk about ``*``-patterns, with some specific special rules that only -apply to specific pattern kinds. - -:: - - *-pattern ::= '_' - -A single-underscore identifier is always an "ignore" pattern. It -matches anything, but does not bind it to a variable. - -:: - - exhaustive-pattern ::= identifier - match-pattern ::= '?' identifier - -Any more complicated identifier is a variable-binding pattern. It is -illegal to bind the same identifier multiple times within a pattern. -However, the variable does come into scope immediately, so in a match -pattern you can have a latter expression which refers to an -already-bound variable. I'm comfortable with constraining this to -only work "conveniently" left-to-right and requiring more complicated -matches to use guard expressions. - -In a match pattern, variable bindings must be prefixed with a ? to -disambiguate them from an expression consisting of a variable -reference. I considered using 'var' instead, but using punctuation -means we don't need a space, which means this is much more compact in -practice. - -Annotation patterns -................... - -:: - - exhaustive-pattern ::= exhaustive-pattern ':' type - -In an exhaustive pattern, you can annotate an arbitrary sub-pattern -with a type. This is useful in an exhaustive pattern: the type of a -variable isn't always inferable (or correctly inferable), and types -in function signatures are generally outright required. It's not as -useful in a match pattern, and the colon can be grammatically awkward -there, so we disallow it. - -'is' patterns -.............. - -:: - - match-pattern ::= 'is' type - -This pattern is satisfied if the dynamic type of the matched value -"satisfies" the named type: - - - if the named type is an Objective-C class type, the dynamic type - must be a class type, and an 'isKindOf:' check is performed; - - - if the named type is a Swift class type, the dynamic type must be - a class type, and a subtype check is performed; - - - if the named type is a metatype, the dynamic type must be a metatype, - and the object type of the dynamic type must satisfy the object type - of the named type; - - - otherwise the named type must equal the dynamic type. - -This inquiry is about dynamic types; archetypes and existentials are -looked through. - -The pattern is ill-formed if it provably cannot be satisfied. - -In a 'switch' statement, this would typically appear like this:: - - case is NSObject: - -It can, however, appear in recursive positions:: - - case (is NSObject, is NSObject): - -Ambiguity with type value matching -`````````````````````````````````` - -There is a potential point of confusion here with dynamic type -checking (done by an 'is' pattern) vs. value equality on type objects -(done by an expression pattern where the expression is of metatype -type. This is resolved by the proposal (currently outstanding but -generally accepted, I think) to disallow naked references to type -constants and instead require them to be somehow decorated. - -That is, this pattern requires the user to write something like this:: - - case is NSObject: - -It is quite likely that users will often accidentally write something -like this:: - - case NSObject: - -It would be very bad if that were actually accepted as a valid -expression but with the very different semantics of testing equality -of type objects. For the most part, type-checking would reject that -as invalid, but a switch on (say) a value of archetype type would -generally work around that. - -However, we have an outstanding proposal to generally forbid 'NSObject' -from appearing as a general expression; the user would have to decorate -it like the following, which would let us eliminate the common mistake:: - - case NSObject.type: - - -Type refinement -``````````````` - -If the value matched is immediately the value of a local variable, I -think it would be really useful if this pattern could introduce a type -refinement within its case, so that the local variable would have the -refined type within that scope. However, making this kind of type -refinement sound would require us to prevent there from being any sort -of mutable alias of the local variable under an unrefined type. -That's usually going to be fine in Swift because we usually don't -permit the address of a local to escape in a way that crosses -statement boundaries. However, closures are a major problem for this -model. If we had immutable local bindings --- and, better yet, if -they were the default --- this problem would largely go away. - -This sort of type refinement could also be a problem with code like:: - - while expr is ParenExpr { - expr = expr.getSubExpr() - } - -It's tricky. - -"Call" patterns -............... - -:: - - match-pattern ::= match-pattern-identifier match-pattern-tuple? - match-pattern-identifier ::= '.' identifier - match-pattern-identifier ::= match-pattern-identifier-tower - match-pattern-identifier-tower ::= identifier - match-pattern-identifier-tower ::= identifier - match-pattern-identifier-tower ::= match-pattern-identifier-tower '.' identifier - -A match pattern can resemble a global name or a call to a global name. -The global name is resolved as normal, and then the pattern is -interpreted according to what is found: - -- If the name resolves to a type, then the dynamic type of the matched - value must match the named type (according to the rules below for - 'is' patterns). It is okay for this to be trivially true. - - In addition, there must be an non-empty arguments clause, and each - element in the clause must have an identifier. For each element, - the identifier must correspond to a known property of the named - type, and the value of that property must satisfy the element - pattern. - -- If the name resolves to a enum element, then the dynamic type - of the matched value must match the enum type as discussed above, - and the value must be of the specified element. There must be - an arguments clause if and only if the element has a value type. - If so, the value of the element is matched against the clause - pattern. - -- Otherwise, the argument clause (if present) must also be - syntactically valid as an expression, and the entire pattern is - reinterpreted as an expression. - -This is all a bit lookup-sensitive, which makes me uncomfortable, but -otherwise I think it makes for attractive syntax. I'm also a little -worried about the way that, say, :code:`f(x)` is always an expression -but :code:`A(x)` is a pattern. Requiring property names when matching -properties goes some way towards making that okay. - -I'm not totally sold on not allowing positional matching against -struct elements; that seems unfortunate in cases where positionality -is conventionally unambiguous, like with a point. - -Matching against struct types requires arguments because this is -intended to be used for structure decomposition, not dynamic type -testing. For the latter, an 'is' pattern should be used. - -Expression patterns -................... - -:: - - match-pattern ::= expression - -When ambiguous, match patterns are interpreted using a -pattern-specific production. I believe it should be true that, in -general, match patterns for a production accept a strict superset of -valid expressions, so that (e.g.) we do not need to disambiguate -whether an open paren starts a tuple expression or a tuple pattern, -but can instead just aggressively parse as a pattern. Note that -binary operators can mean that, using this strategy, we sometimes have -to retroactively rewrite a pattern as an expression. - -It's always possible to disambiguate something as an expression by -doing something not allowing in patterns, like using a unary operator -or calling an identity function; those seem like unfortunate language -solutions, though. - -Satisfying an expression pattern -................................ - -A value satisfies an expression pattern if the match operation -succeeds. I think it would be natural for this match operation to be -spelled the same way as that match-expression operator, so e.g. a -member function called 'matches' or a global binary operator called -'~' or whatever. - -The lookup of this operation poses some interesting questions. In -general, the operation itself is likely to be associated with the -intended type of the expression pattern, but that type will often -require refinement from the type of the matched value. - -For example, consider a pattern like this:: - - case 0...10: - -We should be able to use this pattern when switching on a value which -is not an Int, but if we type-check the expression on its own, we will -assign it the type Range, which will not necessarily permit us -to match (say) a UInt8. - -Order of evaluation of patterns -............................... - -I'd like to keep the order of evaluation and testing of expressions -within a pattern unspecified if I can; I imagine that there should be -a lot of cases where we can rule out a case using a cheap test instead -of a more expensive one, and it would suck to have to run the -expensive one just to have cleaner formal semantics. Specifically, -I'm worried about cases like :code:`case [foo(), 0]:`; if we can test -against 0 before calling :code:`foo()`, that would be great. Also, if -a name is bound and then used directly as an expression later on, it -would be nice to have some flexibility about which value is actually -copied into the variable, but this is less critical. - -:: - - *-pattern ::= *-pattern-tuple - *-pattern-tuple ::= '(' *-pattern-tuple-element-list? '...'? ')' - *-pattern-tuple-element-list ::= *-pattern-tuple-element - *-pattern-tuple-element-list ::= *-pattern-tuple-element ',' pattern-tuple-element-list - *-pattern-tuple-element ::= *-pattern - *-pattern-tuple-element ::= identifier '=' *-pattern - -Tuples are interesting because of the labelled / non-labelled -distinction. Especially with labelled elements, it is really nice to -be able to ignore all the elements you don't care about. This grammar -permits some prefix or set of labels to be matched and the rest to be -ignored. - -Miscellaneous -------------- - -It would be interesting to allow overloading / customization of -pattern-matching. We may find ourselves needing to do something like this to -support non-fragile pattern matching anyway (if there's some set of restrictions -that make it reasonable to permit that). The obvious idea of compiling into the -visitor pattern is a bit compelling, although control flow would be tricky — -we'd probably need the generated code to throw an exception. Alternatively, we -could let the non-fragile type convert itself into a fragile type for purposes -of pattern matching. - -If we ever allow infix ADT constructors, we'll need to allow them in patterns as -well. - -Eventually, we will build regular expressions into the language, and we will -allow them directly as patterns and even bind grouping expressions into user -variables. - -John. diff --git a/docs/SIL.md b/docs/SIL.md new file mode 100644 index 0000000000000..7f40eb45732d4 --- /dev/null +++ b/docs/SIL.md @@ -0,0 +1,3966 @@ +Swift Intermediate Language (SIL) +================================= + +Abstract +-------- + +SIL is an SSA-form IR with high-level semantic information designed to +implement the Swift programming language. SIL accommodates the following +use cases: + +- A set of guaranteed high-level optimizations that provide a + predictable baseline for runtime and diagnostic behavior. +- Diagnostic dataflow analysis passes that enforce Swift language + requirements, such as definitive initialization of variables and + constructors, code reachability, switch coverage. +- High-level optimization passes, including retain/release + optimization, dynamic method devirtualization, closure inlining, + memory allocation promotion, and generic function instantiation. +- A stable distribution format that can be used to distribute + "fragile" inlineable or generic code with Swift library modules, to + be optimized into client binaries. + +In contrast to LLVM IR, SIL is a generally target-independent format +representation that can be used for code distribution, but it can also +express target-specific concepts as well as LLVM can. + +SIL in the Swift Compiler +------------------------- + +At a high level, the Swift compiler follows a strict pipeline +architecture: + +- The *Parse* module constructs an AST from Swift source code. +- The *Sema* module type-checks the AST and annotates it with + type information. +- The *SILGen* module generates *raw SIL* from an AST. +- A series of *Guaranteed Optimization Passes* and *Diagnostic Passes* + are run over the raw SIL both to perform optimizations and to emit + language-specific diagnostics. These are always run, even at -Onone, + and produce *canonical SIL*. +- General SIL *Optimization Passes* optionally run over the canonical + SIL to improve performance of the resulting executable. These are + enabled and controlled by the optimization level and are not run + at -Onone. +- *IRGen* lowers canonical SIL to LLVM IR. +- The LLVM backend (optionally) applies LLVM optimizations, runs the + LLVM code generator and emits binary code. + +The stages pertaining to SIL processing in particular are as follows: + +### SILGen + +SILGen produces *raw SIL* by walking a type-checked Swift AST. The form +of SIL emitted by SILGen has the following properties: + +- Variables are represented by loading and storing mutable memory + locations instead of being in strict SSA form. This is similar to + the initial `alloca`-heavy LLVM IR emitted by frontends such + as Clang. However, Swift represents variables as reference-counted + "boxes" in the most general case, which can be retained, released, + and captured into closures. +- Dataflow requirements, such as definitive assignment, function + returns, switch coverage (TBD), etc. have not yet been enforced. +- `transparent` function optimization has not yet been honored. + +These properties are addressed by subsequent guaranteed optimization and +diagnostic passes which are always run against the raw SIL. + +### Guaranteed Optimization and Diagnostic Passes + +After SILGen, a deterministic sequence of optimization passes is run +over the raw SIL. We do not want the diagnostics produced by the +compiler to change as the compiler evolves, so these passes are intended +to be simple and predictable. + +- **Mandatory inlining** inlines calls to "transparent" functions. +- **Memory promotion** is implemented as two optimization phases, the + first of which performs capture analysis to promote `alloc_box` + instructions to `alloc_stack`, and the second of which promotes + non-address-exposed `alloc_stack` instructions to SSA registers. +- **Constant propagation** folds constant expressions and propagates + the constant values. If an arithmetic overflow occurs during the + constant expression computation, a diagnostic is issued. +- **Return analysis** verifies that each function returns a value on + every code path and doesn't "fall of the end" of its definition, + which is an error. It also issues an error when a `noreturn` + function returns. +- **Critical edge splitting** splits all critical edges from + terminators that don't support arbitrary basic block arguments (all + non cond\_branch terminators). + +If all diagnostic passes succeed, the final result is the *canonical +SIL* for the program. + +TODO: + +- Generic specialization +- Basic ARC optimization for acceptable performance at -Onone. + +### General Optimization Passes + +SIL captures language-specific type information, making it possible to +perform high-level optimizations that are difficult to perform on LLVM +IR. + +- **Generic Specialization** analyzes specialized calls to generic + functions and generates new specialized version of the functions. + Then it rewrites all specialized usages of the generic to a direct + call of the appropriate specialized function. +- **Witness and VTable Devirtualization** for a given type looks up + the associated method from a class's vtable or a types witness table + and replaces the indirect virtual call with a call to the + mapped function. +- **Performance Inlining** +- **Reference Counting Optimizations** +- **Memory Promotion/Optimizations** +- **High-level domain specific optimizations** The swift compiler + implements high-level optimizations on basic Swift containers such + as Array or String. Domain specific optimizations require a defined + interface between the standard library and the optimizer. More + details can be found here: HighLevelSILOptimizations + +Syntax +------ + +SIL is reliant on Swift's type system and declarations, so SIL syntax is +an extension of Swift's. A `.sil` file is a Swift source file with added +SIL definitions. The Swift source is parsed only for its declarations; +Swift `func` bodies (except for nested declarations) and top-level code +are ignored by the SIL parser. In a `.sil` file, there are no implicit +imports; the `swift` and/or `Builtin` standard modules must be imported +explicitly if used. + +Here is an example of a `.sil` file: + + sil_stage canonical + + import Swift + + // Define types used by the SIL function. + + struct Point { + var x : Double + var y : Double + } + + class Button { + func onClick() + func onMouseDown() + func onMouseUp() + } + + // Declare a Swift function. The body is ignored by SIL. + func taxicabNorm(a:Point) -> Double { + return a.x + a.y + } + + // Define a SIL function. + // The name @_T5norms11taxicabNormfT1aV5norms5Point_Sd is the mangled name + // of the taxicabNorm Swift function. + sil @_T5norms11taxicabNormfT1aV5norms5Point_Sd : $(Point) -> Double { + bb0(%0 : $Point): + // func Swift.+(Double, Double) -> Double + %1 = function_ref @_Tsoi1pfTSdSd_Sd + %2 = struct_extract %0 : $Point, #Point.x + %3 = struct_extract %0 : $Point, #Point.y + %4 = apply %1(%2, %3) : $(Double, Double) -> Double + %5 = return %4 : Double + } + + // Define a SIL vtable. This matches dynamically-dispatched method + // identifiers to their implementations for a known static class type. + sil_vtable Button { + #Button.onClick!1: @_TC5norms6Button7onClickfS0_FT_T_ + #Button.onMouseDown!1: @_TC5norms6Button11onMouseDownfS0_FT_T_ + #Button.onMouseUp!1: @_TC5norms6Button9onMouseUpfS0_FT_T_ + } + +### SIL Stage + + decl ::= sil-stage-decl + sil-stage-decl ::= 'sil_stage' sil-stage + + sil-stage ::= 'raw' + sil-stage ::= 'canonical' + +There are different invariants on SIL depending on what stage of +processing has been applied to it. + +- **Raw SIL** is the form produced by SILGen that has not been run + through guaranteed optimizations or diagnostic passes. Raw SIL may + not have a fully-constructed SSA graph. It may contain + dataflow errors. Some instructions may be represented in + non-canonical forms, such as `assign` and `destroy_addr` for + non-address-only values. Raw SIL should not be used for native code + generation or distribution. +- **Canonical SIL** is SIL as it exists after guaranteed optimizations + and diagnostics. Dataflow errors must be eliminated, and certain + instructions must be canonicalized to simpler forms. Performance + optimization and native code generation are derived from this form, + and a module can be distributed containing SIL in this (or later) + forms. + +SIL files declare the processing stage of the included SIL with one of +the declarations `sil_stage raw` or `sil_stage canonical` at top level. +Only one such declaration may appear in a file. + +### SIL Types + + sil-type ::= '$' '*'? generic-parameter-list? type + +SIL types are introduced with the `$` sigil. SIL's type system is +closely related to Swift's, and so the type after the `$` is parsed +largely according to Swift's type grammar. + +#### Type Lowering + +A *formal type* is the type of a value in Swift, such as an expression +result. Swift's formal type system intentionally abstracts over a large +number of representational issues like ownership transfer conventions +and directness of arguments. However, SIL aims to represent most such +implementation details, and so these differences deserve to be reflected +in the SIL type system. *Type lowering* is the process of turning a +formal type into its *lowered type*. + +It is important to be aware that the lowered type of a declaration need +not be the lowered type of the formal type of that declaration. For +example, the lowered type of a declaration reference: + +- will usually be thin, +- will frequently be uncurried, +- may have a non-Swift calling convention, +- may use bridged types in its interface, and +- may use ownership conventions that differ from Swift's + default conventions. + +#### Abstraction Difference + +Generic functions working with values of unconstrained type must +generally work with them indirectly, e.g. by allocating sufficient +memory for them and then passing around pointers to that memory. +Consider a generic function like this: + + func generateArray(n : Int, generator : () -> T) -> T[] + +The function `generator` will be expected to store its result indirectly +into an address passed in an implicit parameter. There's really just no +reasonable alternative when working with a value of arbitrary type: + +- We don't want to generate a different copy of `generateArray` for + every type `T`. +- We don't want to give every type in the language a + common representation. +- We don't want to dynamically construct a call to `generator` + depending on the type `T`. + +But we also don't want the existence of the generic system to force +inefficiencies on non-generic code. For example, we'd like a function of +type `() -> Int` to be able to return its result directly; and yet, +`() -> Int` is a valid substitution of `() -> T`, and a caller of +`generateArray` should be able to pass an arbitrary `() -> Int` in +as the generator. + +Therefore, the representation of a formal type in a generic context may +differ from the representation of a substitution of that formal type. We +call such differences *abstraction differences*. + +SIL's type system is designed to make abstraction differences always +result in differences between SIL types. The goal is that a +properly-abstracted value should be correctly usable at any level of +substitution. + +In order to achieve this, the formal type of a generic entity should +always be lowered using the abstraction pattern of its unsubstituted +formal type. For example, consider the following generic type: + + struct Generator { + var fn : () -> T + } + var intGen : Generator + +`intGen.fn` has the substituted formal type `() -> Int`, which would +normally lower to the type `@callee_owned () -> Int`, i.e. returning its +result directly. But if that type is properly lowered with the pattern +of its unsubstituted type `() -> T`, it becomes +`@callee_owned (@out Int) -> ()`. + +When a type is lowered using the abstraction pattern of an unrestricted +type, it is lowered as if the pattern were replaced with a type sharing +the same structure but replacing all materializable types with fresh +type variables. + +For example, if `g` has type `Generator<(Int,Int) -> Float>`, `g.fn` is +lowered using the pattern `() -> T`, which eventually causes +`(Int,Int) -> Float` to be lowered using the pattern `T`, which is the +same as lowering it with the pattern `U -> V`; the result is that `g.fn` +has the following lowered type: + + @callee_owned () -> @owned @callee_owned (@out Float, @in (Int,Int)) -> ()``. + +As another example, suppose that `h` has type +`Generator<(Int, @inout Int) -> Float>`. Neither `(Int, @inout Int)` nor +`@inout Int` are potential results of substitution because they aren't +materializable, so `h.fn` has the following lowered type: + + @callee_owned () -> @owned @callee_owned (@out Float, @in Int, @inout Int) + +This system has the property that abstraction patterns are preserved +through repeated substitutions. That is, you can consider a lowered type +to encode an abstraction pattern; lowering `T` by `R` is equivalent to +lowering `T` by (`S` lowered by `R`). + +SILGen has procedures for converting values between abstraction +patterns. + +At present, only function and tuple types are changed by abstraction +differences. + +#### Legal SIL Types + +The type of a value in SIL shall be: + +- a loadable legal SIL type, `$T`, +- the address of a legal SIL type, `$*T`, or +- the address of local storage of a legal SIL type, + `$*@local_storage T`. + +A type `T` is a *legal SIL type* if: + +- it is a function type which satisfies the constraints (below) on + function types in SIL, +- it is a tuple type whose element types are legal SIL types, +- it is a legal Swift type that is not a function, tuple, or l-value + type, or +- it is a `@box` containing a legal SIL type. + +Note that types in other recursive positions in the type grammar are +still formal types. For example, the instance type of a metatype or the +type arguments of a generic type are still formal Swift types, not +lowered SIL types. + +#### Address Types + +The *address of T* `$*T` is a pointer to memory containing a value of +any reference or value type `$T`. This can be an internal pointer into a +data structure. Addresses of loadable types can be loaded and stored to +access values of those types. + +Addresses of address-only types (see below) can only be used with +instructions that manipulate their operands indirectly by address, such +as `copy_addr` or `destroy_addr`, or as arguments to functions. It is +illegal to have a value of type `$T` if `T` is address-only. + +Addresses are not reference-counted pointers like class values are. They +cannot be retained or released. + +Address types are not *first-class*: they cannot appear in recursive +positions in type expressions. For example, the type `$**T` is not a +legal type. + +The address of an address cannot be directly taken. `$**T` is not a +representable type. Values of address type thus cannot be allocated, +loaded, or stored (though addresses can of course be loaded from and +stored to). + +Addresses can be passed as arguments to functions if the corresponding +parameter is indirect. They cannot be returned. + +#### Local Storage Types + +The *address of local storage for T* `$*@local_storage T` is a handle to +a stack allocation of a variable of type `$T`. + +For many types, the handle for a stack allocation is simply the +allocated address itself. However, if a type is runtime-sized, the +compiler must emit code to potentially dynamically allocate memory. SIL +abstracts over such differences by using values of local-storage type as +the first result of `alloc_stack` and the operand of `dealloc_stack`. + +Local-storage address types are not *first-class* in the same sense that +address types are not first-class. + +#### Box Types + +Captured local variables and the payloads of `indirect` value types are +stored on the heap. The type `@box T` is a reference-counted type that +references a box containing a mutable value of type `T`. Boxes always +use Swift-native reference counting, so they can be queried for +uniqueness and cast to the `Builtin.NativeObject` type. + +#### Function Types + +Function types in SIL are different from function types in Swift in a +number of ways: + +- A SIL function type may be generic. For example, accessing a generic + function with `function_ref` will give a value of generic + function type. +- A SIL function type declares its conventional treatment of its + context value: + - If it is `@thin`, the function requires no context value. + - If it is `@callee_owned`, the context value is treated as an + owned direct parameter. + - If it is `@callee_guaranteed`, the context value is treated as a + guaranteed direct parameter. + - Otherwise, the context value is treated as an unowned + direct parameter. +- A SIL function type declares the conventions for its parameters, + including any implicit out-parameters. The parameters are written as + an unlabelled tuple; the elements of that tuple must be legal SIL + types, optionally decorated with one of the following + convention attributes. + + The value of an indirect parameter has type `*T`; the value of a + direct parameter has type `T`. + + - An `@in` parameter is indirect. The address must be of an + initialized object; the function is responsible for destroying + the value held there. + - An `@inout` parameter is indirect. The address must be of an + initialized object, and the function must leave an initialized + object there upon exit. + - An `@out` parameter is indirect. The address must be of an + uninitialized object; the function is responsible for + initializing a value there. If there is an `@out` parameter, it + must be the first parameter, and the direct result must be `()`. + - An `@owned` parameter is an owned direct parameter. + - A `@guaranteed` parameter is a guaranteed direct parameter. + - An `@in_guaranteed` parameter is indirect. The address must be + of an initialized object; both the caller and callee promise not + to mutate the pointee, allowing the callee to read it. + - Otherwise, the parameter is an unowned direct parameter. +- A SIL function type declares the convention for its direct result. + The result must be a legal SIL type. + - An `@owned` result is an owned direct result. + - An `@autoreleased` result is an autoreleased direct result. + - Otherwise, the parameter is an unowned direct result. + +A direct parameter or result of trivial type must always be unowned. + +An owned direct parameter or result is transferred to the recipient, +which becomes responsible for destroying the value. This means that the +value is passed at +1. + +An unowned direct parameter or result is instantaneously valid at the +point of transfer. The recipient does not need to worry about race +conditions immediately destroying the value, but should copy it (e.g. by +`strong_retain`ing an object pointer) if the value will be needed sooner +rather than later. + +A guaranteed direct parameter is like an unowned direct parameter value, +except that it is guaranteed by the caller to remain valid throughout +the execution of the call. This means that any `strong_retain`, +`strong_release` pairs in the callee on the argument can be eliminated. + +An autoreleased direct result must have a type with a retainable pointer +representation. It may have been autoreleased, and the caller should +take action to reclaim that autorelease with +`strong_retain_autoreleased`. + +- The @noescape declaration attribute on Swift parameters (which is + valid only on parameters of function type, and is implied by the + @autoclosure attribute) is turned into a @noescape type attribute on + SIL arguments. @noescape indicates that the lifetime of the closure + parameter will not be extended by the callee (e.g. the pointer will + not be stored in a global variable). It corresponds to the LLVM + "nocapture" attribute in terms of semantics (but is limited to only + work with parameters of function type in Swift). +- SIL function types may provide an optional error result, written by + placing `@error` on a result. An error result is always implicitly + `@owned`. Only functions with a native calling convention may have + an error result. + + A function with an error result cannot be called with `apply`. It + must be called with `try_apply`. There is one exception to this + rule: a function with an error result can be called with + `apply [nothrow]` if the compiler can prove that the function does + not actually throw. + + `return` produces a normal result of the function. To return an + error result, use `throw`. + + Type lowering lowers the `throws` annotation on formal function + types into more concrete error propagation: + + - For native Swift functions, `throws` is turned into an + error result. + - For non-native Swift functions, `throws` is turned in an + explicit error-handling mechanism based on the imported API. The + importer only imports non-native methods and types as `throws` + when it is possible to do this automatically. + +#### Properties of Types + +SIL classifies types into additional subgroups based on ABI stability +and generic constraints: + +- *Loadable types* are types with a fully exposed concrete + representation: + + - Reference types + - Builtin value types + - Fragile struct types in which all element types are loadable + - Tuple types in which all element types are loadable + - Class protocol types + - Archetypes constrained by a class protocol + + A *loadable aggregate type* is a tuple or struct type that + is loadable. + + A *trivial type* is a loadable type with trivial value semantics. + Values of trivial type can be loaded and stored without any retain + or release operations and do not need to be destroyed. + +- *Runtime-sized types* are restricted value types for which the + compiler does not know the size of the type statically: + - Resilient value types + - Fragile struct or tuple types that contain resilient types as + elements at any depth + - Archetypes not constrained by a class protocol +- *Address-only types* are restricted value types which cannot be + loaded or otherwise worked with as SSA values: + + - Runtime-sized types + - Non-class protocol types + - @weak types + + Values of address-only type (“address-only values”) must reside in + memory and can only be referenced in SIL by address. Addresses of + address-only values cannot be loaded from or stored to. SIL provides + special instructions for indirectly manipulating address-only + values, such as `copy_addr` and `destroy_addr`. + +Some additional meaningful categories of type: + +- A *heap object reference* type is a type whose representation + consists of a single strong-reference-counted pointer. This includes + all class types, the `Builtin.ObjectPointer` and + `Builtin.ObjCPointer` types, and archetypes that conform to one or + more class protocols. +- A *reference type* is more general in that its low-level + representation may include additional global pointers alongside a + strong-reference-counted pointer. This includes all heap object + reference types and adds thick function types and protocol/protocol + composition types that conform to one or more class protocols. All + reference types can be `retain`-ed and `release`-d. Reference types + also have *ownership semantics* for their referenced heap object; + see Reference Counting\_ below. +- A type with *retainable pointer representation* is guaranteed to be + compatible (in the C sense) with the Objective-C `id` type. The + value at runtime may be `nil`. This includes classes, class + metatypes, block functions, and class-bounded existentials with only + Objective-C-compatible protocol constraints, as well as one level of + `Optional` or `ImplicitlyUnwrappedOptional` applied to any of + the above. Types with retainable pointer representation can be + returned via the `@autoreleased` return convention. + +SILGen does not always map Swift function types one-to-one to SIL +function types. Function types are transformed in order to encode +additional attributes: + +- The **convention** of the function, indicated by the + + attribute. This is similar to the language-level `@convention` + attribute, though SIL extends the set of supported conventions with + additional distinctions not exposed at the language level: + + - `@convention(thin)` indicates a "thin" function reference, which + uses the Swift calling convention with no special "self" or + "context" parameters. + - `@convention(thick)` indicates a "thick" function reference, + which uses the Swift calling convention and carries a + reference-counted context object used to represent captures or + other state required by the function. + - `@convention(block)` indicates an Objective-C compatible + block reference. The function value is represented as a + reference to the block object, which is an `id`-compatible + Objective-C object that embeds its invocation function within + the object. The invocation function uses the C + calling convention. + - `@convention(c)` indicates a C function reference. The function + value carries no context and uses the C calling convention. + - `@convention(objc_method)` indicates an Objective-C + method implementation. The function uses the C calling + convention, with the SIL-level `self` parameter (by SIL + convention mapped to the final formal parameter) mapped to the + `self` and `_cmd` arguments of the implementation. + - `@convention(method)` indicates a Swift instance + method implementation. The function uses the Swift calling + convention, using the special `self` parameter. + - `@convention(witness_method)` indicates a Swift protocol + method implementation. The function's polymorphic convention is + emitted in such a way as to guarantee that it is polymorphic + across all possible implementors of the protocol. +- The **fully uncurried representation** of the function type, with + all of the curried argument clauses flattened into a single + argument clause. For instance, a curried function + `func foo(x:A)(y:B) -> C` might be emitted as a function of type + `((y:B), (x:A)) -> C`. The exact representation depends on the + function's calling + convention\_, which determines the exact ordering of + currying clauses. Methods are treated as a form of curried function. + +#### Layout Compatible Types + +(This section applies only to Swift 1.0 and will hopefully be obviated +in future releases.) + +SIL tries to be ignorant of the details of type layout, and low-level +bit-banging operations such as pointer casts are generally undefined. +However, as a concession to implementation convenience, some types are +allowed to be considered **layout compatible**. Type `T` is *layout +compatible* with type `U` iff: + +- an address of type `$*U` can be cast by + `address_to_pointer`/`pointer_to_address` to `$*T` and a valid value + of type `T` can be loaded out (or indirectly used, if `T` is + address-only), +- if `T` is a nontrivial type, then `retain_value`/`release_value` of + the loaded `T` value is equivalent to `retain_value`/`release_value` + of the original `U` value. + +This is not always a commutative relationship; `T` can be +layout-compatible with `U` whereas `U` is not layout-compatible with +`T`. If the layout compatible relationship does extend both ways, `T` +and `U` are **commutatively layout compatible**. It is however always +transitive; if `T` is layout-compatible with `U` and `U` is +layout-compatible with `V`, then `T` is layout-compatible with `V`. All +types are layout-compatible with themselves. + +The following types are considered layout-compatible: + +- `Builtin.RawPointer` is commutatively layout compatible with all + heap object reference types, and `Optional` of heap object + reference types. (Note that `RawPointer` is a trivial type, so does + not have ownership semantics.) +- `Builtin.RawPointer` is commutatively layout compatible with + `Builtin.Word`. +- Structs containing a single stored property are commutatively layout + compatible with the type of that property. +- A heap object reference is commutatively layout compatible with any + type that can correctly reference the heap object. For instance, + given a class `B` and a derived class `D` inheriting from `B`, a + value of type `B` referencing an instance of type `D` is layout + compatible with both `B` and `D`, as well as `Builtin.NativeObject` + and `Builtin.UnknownObject`. It is not layout compatible with an + unrelated class type `E`. +- For payloaded enums, the payload type of the first payloaded case is + layout-compatible with the enum (*not* commutatively). + +### Values and Operands + + sil-identifier ::= [A-Za-z_0-9]+ + sil-value-name ::= '%' sil-identifier + sil-value ::= sil-value-name ('#' [0-9]+)? + sil-value ::= 'undef' + sil-operand ::= sil-value ':' sil-type + +SIL values are introduced with the `%` sigil and named by an +alphanumeric identifier, which references the instruction or basic block +argument that produces the value. SIL values may also refer to the +keyword 'undef', which is a value of undefined contents. In SIL, a +single instruction may produce multiple values. Operands that refer to +multiple-value instructions choose the value by following the `%name` +with `#` and the index of the value. For example: + + // alloc_box produces two values--the refcounted pointer %box#0, and the + // value address %box#1 + %box = alloc_box $Int64 + // Refer to the refcounted pointer + strong_retain %box#0 : $@box Int64 + // Refer to the address + store %value to %box#1 : $*Int64 + +Unlike LLVM IR, SIL instructions that take value operands *only* accept +value operands. References to literal constants, functions, global +variables, or other entities require specialized instructions such as +`integer_literal`, `function_ref`, `global_addr`, etc. + +### Functions + + decl ::= sil-function + sil-function ::= 'sil' sil-linkage? sil-function-name ':' sil-type + '{' sil-basic-block+ '}' + sil-function-name ::= '@' [A-Za-z_0-9]+ + +SIL functions are defined with the `sil` keyword. SIL function names are +introduced with the `@` sigil and named by an alphanumeric identifier. +This name will become the LLVM IR name for the function, and is usually +the mangled name of the originating Swift declaration. The `sil` syntax +declares the function's name and SIL type, and defines the body of the +function inside braces. The declared type must be a function type, which +may be generic. + +### Basic Blocks + + sil-basic-block ::= sil-label sil-instruction-def* sil-terminator + sil-label ::= sil-identifier ('(' sil-argument (',' sil-argument)* ')')? ':' + sil-argument ::= sil-value-name ':' sil-type + + sil-instruction-def ::= (sil-value-name '=')? sil-instruction + +A function body consists of one or more basic blocks that correspond to +the nodes of the function's control flow graph. Each basic block +contains one or more instructions and ends with a terminator +instruction. The function's entry point is always the first basic block +in its body. + +In SIL, basic blocks take arguments, which are used as an alternative to +LLVM's phi nodes. Basic block arguments are bound by the branch from the +predecessor block: + + sil @iif : $(Builtin.Int1, Builtin.Int64, Builtin.Int64) -> Builtin.Int64 { + bb0(%cond : $Builtin.Int1, %ifTrue : $Builtin.Int64, %ifFalse : $Builtin.Int64): + cond_br %cond : $Builtin.Int1, then, else + then: + br finish(%ifTrue : $Builtin.Int64) + else: + br finish(%ifFalse : $Builtin.Int64) + finish(%result : $Builtin.Int64): + return %result : $Builtin.Int64 + } + +Arguments to the entry point basic block, which has no predecessor, are +bound by the function's caller: + + sil @foo : $(Int) -> Int { + bb0(%x : $Int): + %1 = return %x : $Int + } + + sil @bar : $(Int, Int) -> () { + bb0(%x : $Int, %y : $Int): + %foo = function_ref @foo + %1 = apply %foo(%x) : $(Int) -> Int + %2 = apply %foo(%y) : $(Int) -> Int + %3 = tuple () + %4 = return %3 : $() + } + +### Declaration References + + sil-decl-ref ::= '#' sil-identifier ('.' sil-identifier)* sil-decl-subref? + sil-decl-subref ::= '!' sil-decl-subref-part ('.' sil-decl-uncurry-level)? ('.' sil-decl-lang)? + sil-decl-subref ::= '!' sil-decl-uncurry-level ('.' sil-decl-lang)? + sil-decl-subref ::= '!' sil-decl-lang + sil-decl-subref-part ::= 'getter' + sil-decl-subref-part ::= 'setter' + sil-decl-subref-part ::= 'allocator' + sil-decl-subref-part ::= 'initializer' + sil-decl-subref-part ::= 'enumelt' + sil-decl-subref-part ::= 'destroyer' + sil-decl-subref-part ::= 'deallocator' + sil-decl-subref-part ::= 'globalaccessor' + sil-decl-subref-part ::= 'ivardestroyer' + sil-decl-subref-part ::= 'ivarinitializer' + sil-decl-subref-part ::= 'defaultarg' '.' [0-9]+ + sil-decl-uncurry-level ::= [0-9]+ + sil-decl-lang ::= 'foreign' + +Some SIL instructions need to reference Swift declarations directly. +These references are introduced with the `#` sigil followed by the fully +qualified name of the Swift declaration. Some Swift declarations are +decomposed into multiple entities at the SIL level. These are +distinguished by following the qualified name with `!` and one or more +`.`-separated component entity discriminators: + +- `getter`: the getter function for a `var` declaration +- `setter`: the setter function for a `var` declaration +- `allocator`: a `struct` or `enum` constructor, or a `class`'s + *allocating constructor* +- `initializer`: a `class`'s *initializing constructor* +- `enumelt`: a member of a `enum` type. +- `destroyer`: a class's destroying destructor +- `deallocator`: a class's deallocating destructor +- `globalaccessor`: the addressor function for a global variable +- `ivardestroyer`: a class's ivar destroyer +- `ivarinitializer`: a class's ivar initializer +- `defaultarg.`*n*: the default argument-generating function for the + *n*-th argument of a Swift `func` +- `foreign`: a specific entry point for C/objective-C interoperability + +Methods and curried function definitions in Swift also have multiple +"uncurry levels" in SIL, representing the function at each possible +partial application level. For a curried function declaration: + + // Module example + func foo(x:A)(y:B)(z:C) -> D + +The declaration references and types for the different uncurry levels +are as follows: + + #example.foo!0 : $@thin (x:A) -> (y:B) -> (z:C) -> D + #example.foo!1 : $@thin ((y:B), (x:A)) -> (z:C) -> D + #example.foo!2 : $@thin ((z:C), (y:B), (x:A)) -> D + +The deepest uncurry level is referred to as the **natural uncurry +level**. In this specific example, the reference at the natural uncurry +level is `#example.foo!2`. Note that the uncurried argument clauses are +composed right-to-left, as specified in the calling convention\_. For +uncurry levels less than the uncurry level, the entry point itself is +`@thin` but returns a thick function value carrying the partially +applied arguments for its context. + +Dynamic dispatch\_ instructions such as `class method` require their +method declaration reference to be uncurried to at least uncurry level 1 +(which applies both the "self" argument and the method arguments), +because uncurry level zero represents the application of the method to +its "self" argument, as in `foo.method`, which is where the dynamic +dispatch semantically occurs in Swift. + +### Linkage + + sil-linkage ::= 'public' + sil-linkage ::= 'hidden' + sil-linkage ::= 'shared' + sil-linkage ::= 'private' + sil-linkage ::= 'public_external' + sil-linkage ::= 'hidden_external' + +A linkage specifier controls the situations in which two objects in +different SIL modules are *linked*, i.e. treated as the same object. + +A linkage is *external* if it ends with the suffix `external`. An object +must be a definition if its linkage is not external. + +All functions, global variables, and witness tables have linkage. The +default linkage of a definition is `public`. The default linkage of a +declaration is `public_external`. (These may eventually change to +`hidden` and `hidden_external`, respectively.) + +On a global variable, an external linkage is what indicates that the +variable is not a definition. A variable lacking an explicit linkage +specifier is presumed a definition (and thus gets the default linkage +for definitions, `public`.) + +#### Definition of the *linked* relation + +Two objects are linked if they have the same name and are mutually +visible: + +> - An object with `public` or `public_external` linkage is +> always visible. +> - An object with `hidden`, `hidden_external`, or `shared` linkage is +> visible only to objects in the same Swift module. +> - An object with `private` linkage is visible only to objects in the +> same SIL module. + +Note that the *linked* relationship is an equivalence relation: it is +reflexive, symmetric, and transitive. + +#### Requirements on linked objects + +If two objects are linked, they must have the same type. + +If two objects are linked, they must have the same linkage, except: + +> - A `public` object may be linked to a `public_external` object. +> - A `hidden` object may be linked to a `hidden_external` object. + +If two objects are linked, at most one may be a definition, unless: + +> - both objects have `shared` linkage or +> - at least one of the objects has an external linkage. + +If two objects are linked, and both are definitions, then the +definitions must be semantically equivalent. This equivalence may exist +only on the level of user-visible semantics of well-defined code; it +should not be taken to guarantee that the linked definitions are exactly +operationally equivalent. For example, one definition of a function +might copy a value out of an address parameter, while another may have +had an analysis applied to prove that said value is not needed. + +If an object has any uses, then it must be linked to a definition with +non-external linkage. + +#### Summary + +> - `public` definitions are unique and visible everywhere in +> the program. In LLVM IR, they will be emitted with `external` +> linkage and `default` visibility. +> - `hidden` definitions are unique and visible only within the +> current Swift module. In LLVM IR, they will be emitted with +> `external` linkage and `hidden` visibility. +> - `private` definitions are unique and visible only within the +> current SIL module. In LLVM IR, they will be emitted with +> `private` linkage. +> - `shared` definitions are visible only within the current +> Swift module. They can be linked only with other `shared` +> definitions, which must be equivalent; therefore, they only need +> to be emitted if actually used. In LLVM IR, they will be emitted +> with `linkonce_odr` linkage and `hidden` visibility. +> - `public_external` and `hidden_external` objects always have +> visible definitions somewhere else. If this object nonetheless has +> a definition, it's only for the benefit of optimization +> or analysis. In LLVM IR, declarations will have `external` linkage +> and definitions (if actually emitted as definitions) will have +> `available_externally` linkage. + +### VTables + + decl ::= sil-vtable + sil-vtable ::= 'sil_vtable' identifier '{' sil-vtable-entry* '}' + + sil-vtable-entry ::= sil-decl-ref ':' sil-function-name + +SIL represents dynamic dispatch for class methods using the +class\_method\_, super\_method\_, and dynamic\_method\_ instructions. +The potential destinations for these dispatch operations are tracked in +`sil_vtable` declarations for every class type. The declaration contains +a mapping from every method of the class (including those inherited from +its base class) to the SIL function that implements the method for that +class: + + class A { + func foo() + func bar() + func bas() + } + + sil @A_foo : $@thin (@owned A) -> () + sil @A_bar : $@thin (@owned A) -> () + sil @A_bas : $@thin (@owned A) -> () + + sil_vtable A { + #A.foo!1: @A_foo + #A.bar!1: @A_bar + #A.bas!1: @A_bas + } + + class B : A { + func bar() + } + + sil @B_bar : $@thin (@owned B) -> () + + sil_vtable B { + #A.foo!1: @A_foo + #A.bar!1: @B_bar + #A.bas!1: @A_bas + } + + class C : B { + func bas() + } + + sil @C_bas : $@thin (@owned C) -> () + + sil_vtable C { + #A.foo!1: @A_foo + #A.bar!1: @B_bar + #A.bas!1: @C_bas + } + +Note that the declaration reference in the vtable is to the +least-derived method visible through that class (in the example above, +`B`'s vtable references `A.bar` and not `B.bar`, and `C`'s vtable +references `A.bas` and not `C.bas`). The Swift AST maintains override +relationships between declarations that can be used to look up +overridden methods in the SIL vtable for a derived class (such as +`C.bas` in `C`'s vtable). + +### Witness Tables + + decl ::= sil-witness-table + sil-witness-table ::= 'sil_witness_table' sil-linkage? + normal-protocol-conformance '{' sil-witness-entry* '}' + +SIL encodes the information needed for dynamic dispatch of generic types +into witness tables. This information is used to produce runtime +dispatch tables when generating binary code. It can also be used by SIL +optimizations to specialize generic functions. A witness table is +emitted for every declared explicit conformance. Generic types share one +generic witness table for all of their instances. Derived classes +inherit the witness tables of their base class. + + protocol-conformance ::= normal-protocol-conformance + protocol-conformance ::= 'inherit' '(' protocol-conformance ')' + protocol-conformance ::= 'specialize' '<' substitution* '>' + '(' protocol-conformance ')' + protocol-conformance ::= 'dependent' + normal-protocol-conformance ::= identifier ':' identifier 'module' identifier + +Witness tables are keyed by *protocol conformance*, which is a unique +identifier for a concrete type's conformance to a protocol. + +- A *normal protocol conformance* names a (potentially + unbound generic) type, the protocol it conforms to, and the module + in which the type or extension declaration that provides the + conformance appears. These correspond 1:1 to protocol conformance + declarations in the source code. +- If a derived class conforms to a protocol through inheritance from + its base class, this is represented by an *inherited protocol + conformance*, which simply references the protocol conformance for + the base class. +- If an instance of a generic type conforms to a protocol, it does so + with a *specialized conformance*, which provides the generic + parameter bindings to the normal conformance, which should be for a + generic type. + +Witness tables are only directly associated with normal conformances. +Inherited and specialized conformances indirectly reference the witness +table of the underlying normal conformance. + + sil-witness-entry ::= 'base_protocol' identifier ':' protocol-conformance + sil-witness-entry ::= 'method' sil-decl-ref ':' sil-function-name + sil-witness-entry ::= 'associated_type' identifier + sil-witness-entry ::= 'associated_type_protocol' + '(' identifier ':' identifier ')' ':' protocol-conformance + +Witness tables consist of the following entries: + +- *Base protocol entries* provide references to the protocol + conformances that satisfy the witnessed protocols' + inherited protocols. +- *Method entries* map a method requirement of the protocol to a SIL + function that implements that method for the witness type. One + method entry must exist for every required method of the + witnessed protocol. +- *Associated type entries* map an associated type requirement of the + protocol to the type that satisfies that requirement for the + witness type. Note that the witness type is a source-level Swift + type and not a SIL type. One associated type entry must exist for + every required associated type of the witnessed protocol. +- *Associated type protocol entries* map a protocol requirement on an + associated type to the protocol conformance that satisfies that + requirement for the associated type. + +### Global Variables + + decl ::= sil-global-variable + sil-global-variable ::= 'sil_global' sil-linkage identifier ':' sil-type + +SIL representation of a global variable. + +FIXME: to be written. + +Dataflow Errors +--------------- + +*Dataflow errors* may exist in raw SIL. Swift's semantics defines these +conditions as errors, so they must be diagnosed by diagnostic passes and +must not exist in canonical SIL. + +### Definitive Initialization + +Swift requires that all local variables be initialized before use. In +constructors, all instance variables of a struct, enum, or class type +must be initialized before the object is used and before the constructor +is returned from. + +### Unreachable Control Flow + +The `unreachable` terminator is emitted in raw SIL to mark incorrect +control flow, such as a non-`Void` function failing to `return` a value, +or a `switch` statement failing to cover all possible values of its +subject. The guaranteed dead code elimination pass can eliminate truly +unreachable basic blocks, or `unreachable` instructions may be dominated +by applications of `@noreturn` functions. An `unreachable` instruction +that survives guaranteed DCE and is not immediately preceded by a +`@noreturn` application is a dataflow error. + +Runtime Failure +--------------- + +Some operations, such as failed unconditional checked conversions\_ or +the `Builtin.trap` compiler builtin, cause a *runtime failure*, which +unconditionally terminates the current actor. If it can be proven that a +runtime failure will occur or did occur, runtime failures may be +reordered so long as they remain well-ordered relative to operations +external to the actor or the program as a whole. For instance, with +overflow checking on integer arithmetic enabled, a simple `for` loop +that reads inputs in from one or more arrays and writes outputs to +another array, all local to the current actor, may cause runtime failure +in the update operations: + + // Given unknown start and end values, this loop may overflow + for var i = unknownStartValue; i != unknownEndValue; ++i { + ... + } + +It is permitted to hoist the overflow check and associated runtime +failure out of the loop itself and check the bounds of the loop prior to +entering it, so long as the loop body has no observable effect outside +of the current actor. + +Undefined Behavior +------------------ + +Incorrect use of some operations is *undefined behavior*, such as +invalid unchecked casts involving `Builtin.RawPointer` types, or use of +compiler builtins that lower to LLVM instructions with undefined +behavior at the LLVM level. A SIL program with undefined behavior is +meaningless, much like undefined behavior in C, and has no predictable +semantics. Undefined behavior should not be triggered by valid SIL +emitted by a correct Swift program using a correct standard library, but +cannot in all cases be diagnosed or verified at the SIL level. + +Calling Convention +------------------ + +This section describes how Swift functions are emitted in SIL. + +### Swift Calling Convention @cc(swift) + +The Swift calling convention is the one used by default for native Swift +functions. + +Tuples in the input type of the function are recursively destructured +into separate arguments, both in the entry point basic block of the +callee, and in the `apply` instructions used by callers: + + func foo(x:Int, y:Int) + + sil @foo : $(x:Int, y:Int) -> () { + entry(%x : $Int, %y : $Int): + ... + } + + func bar(x:Int, y:(Int, Int)) + + sil @bar : $(x:Int, y:(Int, Int)) -> () { + entry(%x : $Int, %y0 : $Int, %y1 : $Int): + ... + } + + func call_foo_and_bar() { + foo(1, 2) + bar(4, (5, 6)) + } + + sil @call_foo_and_bar : $() -> () { + entry: + ... + %foo = function_ref @foo : $(x:Int, y:Int) -> () + %foo_result = apply %foo(%1, %2) : $(x:Int, y:Int) -> () + ... + %bar = function_ref @bar : $(x:Int, y:(Int, Int)) -> () + %bar_result = apply %bar(%4, %5, %6) : $(x:Int, y:(Int, Int)) -> () + } + +Calling a function with trivial value types as inputs and outputs simply +passes the arguments by value. This Swift function: + + func foo(x:Int, y:Float) -> UnicodeScalar + + foo(x, y) + +gets called in SIL as: + + %foo = constant_ref $(Int, Float) -> UnicodeScalar, @foo + %z = apply %foo(%x, %y) : $(Int, Float) -> UnicodeScalar + +#### Reference Counts + +*NOTE* This section only is speaking in terms of rules of thumb. The +actual behavior of arguments with respect to arguments is defined by the +argument's convention attribute (e.g. `@owned`), not the calling +convention itself. + +Reference type arguments are passed in at +1 retain count and consumed +by the callee. A reference type return value is returned at +1 and +consumed by the caller. Value types with reference type components have +their reference type components each retained and released the same way. +This Swift function: + + class A {} + + func bar(x:A) -> (Int, A) { ... } + + bar(x) + +gets called in SIL as: + + %bar = function_ref @bar : $(A) -> (Int, A) + strong_retain %x : $A + %z = apply %bar(%x) : $(A) -> (Int, A) + // ... use %z ... + %z_1 = tuple_extract %z : $(Int, A), 1 + strong_release %z_1 + +When applying a thick function value as a callee, the function value is +also consumed at +1 retain count. + +#### Address-Only Types + +For address-only arguments, the caller allocates a copy and passes the +address of the copy to the callee. The callee takes ownership of the +copy and is responsible for destroying or consuming the value, though +the caller must still deallocate the memory. For address-only return +values, the caller allocates an uninitialized buffer and passes its +address as the first argument to the callee. The callee must initialize +this buffer before returning. This Swift function: + + @API struct A {} + +> func bas(x:A, y:Int) -> A { return x } +> +> var z = bas(x, y) // ... use z ... + +gets called in SIL as: + + %bas = function_ref @bas : $(A, Int) -> A + %z = alloc_stack $A + %x_arg = alloc_stack $A + copy_addr %x to [initialize] %x_arg : $*A + apply %bas(%z, %x_arg, %y) : $(A, Int) -> A + dealloc_stack %x_arg : $*A // callee consumes %x.arg, caller deallocs + // ... use %z ... + destroy_addr %z : $*A + dealloc_stack stack %z : $*A + +The implementation of `@bas` is then responsible for consuming `%x_arg` +and initializing `%z`. + +Tuple arguments are destructured regardless of the address-only-ness of +the tuple type. The destructured fields are passed individually +according to the above convention. This Swift function: + + @API struct A {} + + func zim(x:Int, y:A, (z:Int, w:(A, Int))) + + zim(x, y, (z, w)) + +gets called in SIL as: + + %zim = function_ref @zim : $(x:Int, y:A, (z:Int, w:(A, Int))) -> () + %y_arg = alloc_stack $A + copy_addr %y to [initialize] %y_arg : $*A + %w_0_addr = element_addr %w : $*(A, Int), 0 + %w_0_arg = alloc_stack $A + copy_addr %w_0_addr to [initialize] %w_0_arg : $*A + %w_1_addr = element_addr %w : $*(A, Int), 1 + %w_1 = load %w_1_addr : $*Int + apply %zim(%x, %y_arg, %z, %w_0_arg, %w_1) : $(x:Int, y:A, (z:Int, w:(A, Int))) -> () + dealloc_stack %w_0_arg + dealloc_stack %y_arg + +#### Variadic Arguments + +Variadic arguments and tuple elements are packaged into an array and +passed as a single array argument. This Swift function: + + func zang(x:Int, (y:Int, z:Int...), v:Int, w:Int...) + + zang(x, (y, z0, z1), v, w0, w1, w2) + +gets called in SIL as: + + %zang = function_ref @zang : $(x:Int, (y:Int, z:Int...), v:Int, w:Int...) -> () + %zs = <> + %ws = <> + apply %zang(%x, %y, %zs, %v, %ws) : $(x:Int, (y:Int, z:Int...), v:Int, w:Int...) -> () + +#### Function Currying + +Curried function definitions in Swift emit multiple SIL entry points, +one for each "uncurry level" of the function. When a function is +uncurried, its outermost argument clauses are combined into a tuple in +right-to-left order. For the following declaration: + + func curried(x:A)(y:B)(z:C)(w:D) -> Int {} + +The types of the SIL entry points are as follows: + + sil @curried_0 : $(x:A) -> (y:B) -> (z:C) -> (w:D) -> Int { ... } + sil @curried_1 : $((y:B), (x:A)) -> (z:C) -> (w:D) -> Int { ... } + sil @curried_2 : $((z:C), (y:B), (x:A)) -> (w:D) -> Int { ... } + sil @curried_3 : $((w:D), (z:C), (y:B), (x:A)) -> Int { ... } + +#### @inout Arguments + +`@inout` arguments are passed into the entry point by address. The +callee does not take ownership of the referenced memory. The referenced +memory must be initialized upon function entry and exit. If the `@inout` +argument refers to a fragile physical variable, then the argument is the +address of that variable. If the `@inout` argument refers to a logical +property, then the argument is the address of a caller-owned writeback +buffer. It is the caller's responsibility to initialize the buffer by +storing the result of the property getter prior to calling the function +and to write back to the property on return by loading from the buffer +and invoking the setter with the final value. This Swift function: + + func inout(x:@inout Int) { + x = 1 + } + +gets lowered to SIL as: + + sil @inout : $(@inout Int) -> () { + entry(%x : $*Int): + %1 = integer_literal 1 : $Int + store %1 to %x + return + } + +### Swift Method Calling Convention @cc(method) + +The method calling convention is currently identical to the freestanding +function convention. Methods are considered to be curried functions, +taking the "self" argument as their outer argument clause, and the +method arguments as the inner argument clause(s). When uncurried, the +"self" argument is thus passed last: + + struct Foo { + func method(x:Int) -> Int {} + } + + sil @Foo_method_1 : $((x : Int), @inout Foo) -> Int { ... } + +### Witness Method Calling Convention @cc(witness\_method) + +The witness method calling convention is used by protocol witness +methods in witness tables\_. It is identical to the `method` calling +convention except that its handling of generic type parameters. For +non-witness methods, the machine-level convention for passing type +parameter metadata may be arbitrarily dependent on static aspects of the +function signature, but because witnesses must be polymorphically +dispatchable on their `Self` type, the `Self`-related metadata for a +witness must be passed in a maximally abstracted manner. + +### C Calling Convention @cc(cdecl) + +In Swift's C module importer, C types are always mapped to Swift types +considered trivial by SIL. SIL does not concern itself with platform ABI +requirements for indirect return, register vs. stack passing, etc.; C +function arguments and returns in SIL are always by value regardless of +the platform calling convention. + +SIL (and therefore Swift) cannot currently invoke variadic C functions. + +### Objective-C Calling Convention @cc(objc\_method) + +#### Reference Counts + +Objective-C methods use the same argument and return value ownership +rules as ARC Objective-C. Selector families and the `ns_consumed`, +`ns_returns_retained`, etc. attributes from imported Objective-C +definitions are honored. + +Applying a `@convention(block)` value does not consume the block. + +#### Method Currying + +In SIL, the "self" argument of an Objective-C method is uncurried to the +last argument of the uncurried type, just like a native Swift method: + + @objc class NSString { + func stringByPaddingToLength(Int) withString(NSString) startingAtIndex(Int) + } + + sil @NSString_stringByPaddingToLength_withString_startingAtIndex \ + : $((Int, NSString, Int), NSString) + +That `self` is passed as the first argument at the IR level is +abstracted away in SIL, as is the existence of the `_cmd` selector +argument. + +Type Based Alias Analysis +------------------------- + +SIL supports two types of Type Based Alias Analysis (TBAA): Class TBAA +and Typed Access TBAA. + +### Class TBAA + +Class instances and other *heap object references* are pointers at the +implementation level, but unlike SIL addresses, they are first class +values and can be `capture`-d and alias. Swift, however, is memory-safe +and statically typed, so aliasing of classes is constrained by the type +system as follows: + +- A `Builtin.NativeObject` may alias any native Swift heap object, + including a Swift class instance, a box allocated by `alloc_box`, or + a thick function's closure context. It may not alias natively + Objective-C class instances. +- A `Builtin.UnknownObject` may alias any class instance, whether + Swift or Objective-C, but may not alias non-class-instance + heap objects. +- Two values of the same class type `$C` may alias. Two values of + related class type `$B` and `$D`, where there is a subclass + relationship between `$B` and `$D`, may alias. Two values of + unrelated class types may not alias. This includes different + instantiations of a generic class type, such as `$C` and + `$C`, which currently may never alias. +- Without whole-program visibility, values of archetype or protocol + type must be assumed to potentially alias any class instance. Even + if it is locally apparent that a class does not conform to that + protocol, another component may introduce a conformance by + an extension. Similarly, a generic class instance, such as `$C` + for archetype `T`, must be assumed to potentially alias concrete + instances of the generic type, such as `$C`, because `Int` is a + potential substitution for `T`. + +### Typed Access TBAA + +Define a *typed access* of an address or reference as one of the +following: + +- Any instruction that performs a typed read or write operation upon + the memory at the given location (e.x. `load`, `store`). +- Any instruction that yields a typed offset of the pointer by + performing a typed projection operation (e.x. `ref_element_addr`, + `tuple_element_addr`). + +It is undefined behavior to perform a typed access to an address or +reference if the stored object or referent is not an allocated object of +the relevant type. + +This allows the optimizer to assume that two addresses cannot alias if +there does not exist a substitution of archetypes that could cause one +of the types to be the type of a subobject of the other. Additionally, +this applies to the types of the values from which the addresses were +derived, ignoring "blessed" alias-introducing operations such as +`pointer_to_address`, the `bitcast` intrinsic, and the `inttoptr` +intrinsic. + +Value Dependence +---------------- + +In general, analyses can assume that independent values are +independently assured of validity. For example, a class method may +return a class reference: + + bb0(%0 : $MyClass): + %1 = class_method %0 : $MyClass, #MyClass.foo!1 + %2 = apply %1(%0) : $@cc(method) @thin (@guaranteed MyClass) -> @owned MyOtherClass + // use of %2 goes here; no use of %1 + strong_release %2 : $MyOtherClass + strong_release %1 : $MyClass + +The optimizer is free to move the release of `%1` to immediately after +the call here, because `%2` can be assumed to be an +independently-managed value, and because Swift generally permits the +reordering of destructors. + +However, some instructions do create values that are intrinsically +dependent on their operands. For example, the result of +`ref_element_addr` will become a dangling pointer if the base is +released too soon. This is captured by the concept of *value +dependence*, and any transformation which can reorder of destruction of +a value around another operation must remain conscious of it. + +A value `%1` is said to be *value-dependent* on a value `%0` if: + +- `%1` is the result and `%0` is the first operand of one of the + following instructions: + - `ref_element_addr` + - `struct_element_addr` + - `tuple_element_addr` + - `unchecked_take_enum_data_addr` + - `pointer_to_address` + - `address_to_pointer` + - `index_addr` + - `index_raw_pointer` + - possibly some other conversions +- `%1` is the result of `mark_dependence` and `%0` is either of + the operands. +- `%1` is the value address of an allocation instruction of which `%0` + is the local storage token or box reference. +- `%1` is the result of a `struct`, `tuple`, or `enum` instruction and + `%0` is an operand. +- `%1` is the result of projecting out a subobject of `%0` with + `tuple_extract`, `struct_extract`, `unchecked_enum_data`, + `select_enum`, or `select_enum_addr`. +- `%1` is the result of `select_value` and `%0` is one of the cases. +- `%1` is a basic block parameter and `%0` is the corresponding + argument from a branch to that block. +- `%1` is the result of a `load` from `%0`. However, the value + dependence is cut after the first attempt to manage the value of + `%1`, e.g. by retaining it. +- Transitivity: there exists a value `%2` which `%1` depends on and + which depends on `%0`. However, transitivity does not apply to + different subobjects of a struct, tuple, or enum. + +Note, however, that an analysis is not required to track dependence +through memory. Nor is it required to consider the possibility of +dependence being established "behind the scenes" by opaque code, such as +by a method returning an unsafe pointer to a class property. The +dependence is required to be locally obvious in a function's SIL +instructions. Precautions must be taken against this either by SIL +generators (by using `mark_dependence` appropriately) or by the user (by +using the appropriate intrinsics and attributes with unsafe language or +library features). + +Only certain types of SIL value can carry value-dependence: + +- SIL address types +- unmanaged pointer types: + - `@sil_unmanaged` types + - `Builtin.RawPointer` + - aggregates containing such a type, such as `UnsafePointer`, + possibly recursively +- non-trivial types (but they can be independently managed) + +This rule means that casting a pointer to an integer type breaks +value-dependence. This restriction is necessary so that reading an `Int` +from a class doesn't force the class to be kept around! A class holding +an unsafe reference to an object must use some sort of unmanaged pointer +type to do so. + +This rule does not include generic or resilient value types which might +contain unmanaged pointer types. Analyses are free to assume that e.g. a +`copy_addr` of a generic or resilient value type yields an +independently-managed value. The extension of value dependence to types +containing obvious unmanaged pointer types is an affordance to make the +use of such types more convenient; it does not shift the ultimate +responsibility for assuring the safety of unsafe language/library +features away from the user. + +Instruction Set +--------------- + +### Allocation and Deallocation + +These instructions allocate and deallocate memory. + +#### alloc\_stack + + sil-instruction ::= 'alloc_stack' sil-type + + %1 = alloc_stack $T + // %1#0 has type $*@local_storage T + // %1#1 has type $*T + +Allocates uninitialized memory that is sufficiently aligned on the stack +to contain a value of type `T`. The first result of the instruction is a +local-storage handle suitable for passing to `dealloc_stack`. The second +result of the instruction is the address of the allocated memory. + +`alloc_stack` marks the start of the lifetime of the value; the +allocation must be balanced with a `dealloc_stack` instruction to mark +the end of its lifetime. All `alloc_stack` allocations must be +deallocated prior to returning from a function. If a block has multiple +predecessors, the stack height and order of allocations must be +consistent coming from all predecessor blocks. `alloc_stack` allocations +must be deallocated in last-in, first-out stack order. + +The memory is not retainable. To allocate a retainable box for a value +type, use `alloc_box`. + +#### alloc\_ref + + sil-instruction ::= 'alloc_ref' ('[' 'objc' ']')? ('[' 'stack' ']')? sil-type + + %1 = alloc_ref [stack] $T + // $T must be a reference type + // %1 has type $T + +Allocates an object of reference type `T`. The object will be +initialized with retain count 1; its state will be otherwise +uninitialized. The optional `objc` attribute indicates that the object +should be allocated using Objective-C's allocation methods +(`+allocWithZone:`). The optional `stack` attribute indicates that the +object can be allocated on the stack instead on the heap. In this case +the instruction must have balanced with a `dealloc_ref [stack]` +instruction to mark the end of the object's lifetime. Note that the +`stack` attribute only specifies that stack allocation is possible. The +final decision on stack allocation is done during llvm IR generation. +This is because the decision also depends on the object size, which is +not necessarily known at SIL level. + +#### alloc\_ref\_dynamic + + sil-instruction ::= 'alloc_ref_dynamic' ('[' 'objc' ']')? sil-operand ',' sil-type + + %1 = alloc_ref_dynamic %0 : $@thick T.Type, $T + %1 = alloc_ref_dynamic [objc] %0 : $@objc_metatype T.Type, $T + // $T must be a class type + // %1 has type $T + +Allocates an object of class type `T` or a subclass thereof. The dynamic +type of the resulting object is specified via the metatype value `%0`. +The object will be initialized with retain count 1; its state will be +otherwise uninitialized. The optional `objc` attribute indicates that +the object should be allocated using Objective-C's allocation methods +(`+allocWithZone:`). + +#### alloc\_box + + sil-instruction ::= 'alloc_box' sil-type + + %1 = alloc_box $T + // %1 has two values: + // %1#0 has type $@box T + // %1#1 has type $*T + +Allocates a reference-counted `@box` on the heap large enough to hold a +value of type `T`, along with a retain count and any other metadata +required by the runtime. The result of the instruction is a two-value +operand; the first value is the reference-counted `@box` reference that +owns the box, and the second value is the address of the value inside +the box. + +The box will be initialized with a retain count of 1; the storage will +be uninitialized. The box owns the contained value, and releasing it to +a retain count of zero destroys the contained value as if by +`destroy_addr`. Releasing a box is undefined behavior if the box's value +is uninitialized. To deallocate a box whose value has not been +initialized, `dealloc_box` should be used. + +#### alloc\_value\_buffer + + sil-instruction ::= 'alloc_value_buffer' sil-type 'in' sil-operand + + %1 = alloc_value_buffer $(Int, T) in %0 : $*Builtin.UnsafeValueBuffer + // The operand must have the exact type shown. + // The result has type $*(Int, T). + +Given the address of an unallocated value buffer, allocate space in it +for a value of the given type. This instruction has undefined behavior +if the value buffer is currently allocated. + +The type operand must be a lowered object type. + +#### dealloc\_stack + + sil-instruction ::= 'dealloc_stack' sil-operand + + dealloc_stack %0 : $*@local_storage T + // %0 must be of a local-storage $*@local_storage T type + +Deallocates memory previously allocated by `alloc_stack`. The allocated +value in memory must be uninitialized or destroyed prior to being +deallocated. This instruction marks the end of the lifetime for the +value created by the corresponding `alloc_stack` instruction. The +operand must be the `@local_storage` of the shallowest live +`alloc_stack` allocation preceding the deallocation. In other words, +deallocations must be in last-in, first-out stack order. + +#### dealloc\_box + + sil-instruction ::= 'dealloc_box' sil-operand + + dealloc_box %0 : $@box T + +Deallocates a box, bypassing the reference counting mechanism. The box +variable must have a retain count of one. The boxed type must match the +type passed to the corresponding `alloc_box` exactly, or else undefined +behavior results. + +This does not destroy the boxed value. The contents of the value must +have been fully uninitialized or destroyed before `dealloc_box` is +applied. + +#### project\_box + + sil-instruction ::= 'project_box' sil-operand + + %1 = project_box %0 : $@box T + + // %1 has type $*T + +Given a `@box T` reference, produces the address of the value inside the +box. + +#### dealloc\_ref + + sil-instruction ::= 'dealloc_ref' ('[' 'stack' ']')? sil-operand + + dealloc_ref [stack] %0 : $T + // $T must be a class type + +Deallocates an uninitialized class type instance, bypassing the +reference counting mechanism. + +The type of the operand must match the allocated type exactly, or else +undefined behavior results. + +The instance must have a retain count of one. + +This does not destroy stored properties of the instance. The contents of +stored properties must be fully uninitialized at the time `dealloc_ref` +is applied. + +The `stack` attribute indicates that the instruction is the balanced +deallocation of its operand which must be a `alloc_ref [stack]`. In this +case the instruction marks the end of the object's lifetime but has no +other effect. + +#### dealloc\_partial\_ref + + sil-instruction ::= 'dealloc_partial_ref' sil-operand sil-metatype + + dealloc_partial_ref %0 : $T, %1 : $U.Type + // $T must be a class type + // $T must be a subclass of U + +Deallocates a partially-initialized class type instance, bypassing the +reference counting mechanism. + +The type of the operand must be a supertype of the allocated type, or +else undefined behavior results. + +The instance must have a retain count of one. + +All stored properties in classes more derived than the given metatype +value must be initialized, and all other stored properties must be +uninitialized. The initialized stored properties are destroyed before +deallocating the memory for the instance. + +This does not destroy the reference type instance. The contents of the +heap object must have been fully uninitialized or destroyed before +`dealloc_ref` is applied. + +#### dealloc\_value\_buffer + + sil-instruction ::= 'dealloc_value_buffer' sil-type 'in' sil-operand + + dealloc_value_buffer $(Int, T) in %0 : $*Builtin.UnsafeValueBuffer + // The operand must have the exact type shown. + +Given the address of a value buffer, deallocate the storage in it. This +instruction has undefined behavior if the value buffer is not currently +allocated, or if it was allocated with a type other than the type +operand. + +The type operand must be a lowered object type. + +#### project\_value\_buffer + + sil-instruction ::= 'project_value_buffer' sil-type 'in' sil-operand + + %1 = project_value_buffer $(Int, T) in %0 : $*Builtin.UnsafeValueBuffer + // The operand must have the exact type shown. + // The result has type $*(Int, T). + +Given the address of a value buffer, return the address of the value +storage in it. This instruction has undefined behavior if the value +buffer is not currently allocated, or if it was allocated with a type +other than the type operand. + +The result is the same value as was originally returned by +`alloc_value_buffer`. + +The type operand must be a lowered object type. + +### Debug Information + +Debug information is generally associated with allocations (alloc\_stack +or alloc\_box) by having a Decl node attached to the allocation with a +SILLocation. For declarations that have no allocation we have explicit +instructions for doing this. This is used by 'let' declarations, which +bind a value to a name and for var decls who are promoted into +registers. The decl they refer to is attached to the instruction with a +SILLocation. + +#### debug\_value + + sil-instruction ::= debug_value sil-operand + + debug_value %1 : $Int + +This indicates that the value of a declaration with loadable type has +changed value to the specified operand. The declaration in question is +identified by the SILLocation attached to the debug\_value instruction. + +The operand must have loadable type. + +#### debug\_value\_addr + + sil-instruction ::= debug_value_addr sil-operand + + debug_value_addr %7 : $*SomeProtocol + +This indicates that the value of a declaration with address-only type +has changed value to the specified operand. The declaration in question +is identified by the SILLocation attached to the debug\_value\_addr +instruction. + +### Accessing Memory + +#### load + + sil-instruction ::= 'load' sil-operand + + %1 = load %0 : $*T + // %0 must be of a $*T address type for loadable type $T + // %1 will be of type $T + +Loads the value at address `%0` from memory. `T` must be a loadable +type. This does not affect the reference count, if any, of the loaded +value; the value must be retained explicitly if necessary. It is +undefined behavior to load from uninitialized memory or to load from an +address that points to deallocated storage. + +#### store + + sil-instruction ::= 'store' sil-value 'to' sil-operand + + store %0 to %1 : $*T + // $T must be a loadable type + +Stores the value `%0` to memory at address `%1`. The type of %1 is `*T` +and the type of +`%0 is`T`, which must be a loadable type. This will overwrite the memory at`%1`. If`%1`already references a value that requires`release`or other cleanup, that value must be loaded before being stored over and cleaned up. It is undefined behavior to store to an address that points to deallocated storage. assign``` +: + + sil-instruction ::= 'assign' sil-value 'to' sil-operand + + assign %0 to %1 : $*T + // $T must be a loadable type + +Represents an abstract assignment of the value `%0` to memory at address +`%1` without specifying whether it is an initialization or a normal +store. The type of %1 is `*T` and the type of `%0` is `T`, which must be +a loadable type. This will overwrite the memory at `%1` and destroy the +value currently held there. + +The purpose of the `assign` instruction is to simplify the definitive +initialization analysis on loadable variables by removing what would +otherwise appear to be a load and use of the current value. It is +produced by SILGen, which cannot know which assignments are meant to be +initializations. If it is deemed to be an initialization, it can be +replaced with a `store`; otherwise, it must be replaced with a sequence +that also correctly destroys the current value. + +This instruction is only valid in Raw SIL and is rewritten as +appropriate by the definitive initialization pass. + +#### mark\_uninitialized + + sil-instruction ::= 'mark_uninitialized' '[' mu_kind ']' sil-operand + mu_kind ::= 'var' + mu_kind ::= 'rootself' + mu_kind ::= 'derivedself' + mu_kind ::= 'derivedselfonly' + mu_kind ::= 'delegatingself' + + %2 = mark_uninitialized [var] %1 : $*T + // $T must be an address + +Indicates that a symbolic memory location is uninitialized, and must be +explicitly initialized before it escapes or before the current function +returns. This instruction returns its operands, and all accesses within +the function must be performed against the return value of the +mark\_uninitialized instruction. + +The kind of mark\_uninitialized instruction specifies the type of data +the mark\_uninitialized instruction refers to: + +- `var`: designates the start of a normal variable live range +- `rootself`: designates `self` in a struct, enum, or root class +- `derivedself`: designates `self` in a derived (non-root) class +- `derivedselfonly`: designates `self` in a derived (non-root) class + whose stored properties have already been initialized +- `delegatingself`: designates `self` on a struct, enum, or class in a + delegating constructor (one that calls self.init) + +The purpose of the `mark_uninitialized` instruction is to enable +definitive initialization analysis for global variables (when marked as +'globalvar') and instance variables (when marked as 'rootinit'), which +need to be distinguished from simple allocations. + +It is produced by SILGen, and is only valid in Raw SIL. It is rewritten +as appropriate by the definitive initialization pass. + +#### mark\_function\_escape + + sil-instruction ::= 'mark_function_escape' sil-operand (',' sil-operand) + + %2 = mark_function_escape %1 : $*T + +Indicates that a function definition closes over a symbolic memory +location. This instruction is variadic, and all of its operands must be +addresses. + +The purpose of the `mark_function_escape` instruction is to enable +definitive initialization analysis for global variables and instance +variables, which are not represented as box allocations. + +It is produced by SILGen, and is only valid in Raw SIL. It is rewritten +as appropriate by the definitive initialization pass. + +#### copy\_addr + + sil-instruction ::= 'copy_addr' '[take]'? sil-value + 'to' '[initialization]'? sil-operand + + copy_addr [take] %0 to [initialization] %1 : $*T + // %0 and %1 must be of the same $*T address type + +Loads the value at address `%0` from memory and assigns a copy of it +back into memory at address `%1`. A bare `copy_addr` instruction when +`T` is a non-trivial type: + + copy_addr %0 to %1 : $*T + +is equivalent to: + + %new = load %0 : $*T // Load the new value from the source + %old = load %1 : $*T // Load the old value from the destination + strong_retain %new : $T // Retain the new value + strong_release %old : $T // Release the old + store %new to %1 : $*T // Store the new value to the destination + +except that `copy_addr` may be used even if `%0` is of an address-only +type. The `copy_addr` may be given one or both of the `[take]` or +`[initialization]` attributes: + +- `[take]` destroys the value at the source address in the course of + the copy. +- `[initialization]` indicates that the destination address + is uninitialized. Without the attribute, the destination address is + treated as already initialized, and the existing value will be + destroyed before the new value is stored. + +The three attributed forms thus behave like the following loadable type +operations: + + // take-assignment + copy_addr [take] %0 to %1 : $*T + // is equivalent to: + %new = load %0 : $*T + %old = load %1 : $*T + // no retain of %new! + strong_release %old : $T + store %new to %1 : $*T + + // copy-initialization + copy_addr %0 to [initialization] %1 : $*T + // is equivalent to: + %new = load %0 : $*T + strong_retain %new : $T + // no load/release of %old! + store %new to %1 : $*T + + // take-initialization + copy_addr [take] %0 to [initialization] %1 : $*T + // is equivalent to: + %new = load %0 : $*T + // no retain of %new! + // no load/release of %old! + store %new to %1 : $*T + +If `T` is a trivial type, then `copy_addr` is always equivalent to its +take-initialization form. + +#### destroy\_addr + + sil-instruction ::= 'destroy_addr' sil-operand + + destroy_addr %0 : $*T + // %0 must be of an address $*T type + +Destroys the value in memory at address `%0`. If `T` is a non-trivial +type, This is equivalent to: + + %1 = load %0 + strong_release %1 + +except that `destroy_addr` may be used even if `%0` is of an +address-only type. This does not deallocate memory; it only destroys the +pointed-to value, leaving the memory uninitialized. + +If `T` is a trivial type, then `destroy_addr` is a no-op. + +#### index\_addr + + sil-instruction ::= 'index_addr' sil-operand ',' sil-operand + + %2 = index_addr %0 : $*T, %1 : $Builtin.Int + // %0 must be of an address type $*T + // %1 must be of a builtin integer type + // %2 will be of type $*T + +Given an address that references into an array of values, returns the +address of the `%1`-th element relative to `%0`. The address must +reference into a contiguous array. It is undefined to try to reference +offsets within a non-array value, such as fields within a homogeneous +struct or tuple type, or bytes within a value, using `index_addr`. +(`Int8` address types have no special behavior in this regard, unlike +`char*` or `void*` in C.) It is also undefined behavior to index out of +bounds of an array, except to index the "past-the-end" address of the +array. + +#### index\_raw\_pointer + + sil-instruction ::= 'index_raw_pointer' sil-operand ',' sil-operand + + %2 = index_raw_pointer %0 : $Builtin.RawPointer, %1 : $Builtin.Int + // %0 must be of $Builtin.RawPointer type + // %1 must be of a builtin integer type + // %2 will be of type $*T + +Given a `Builtin.RawPointer` value `%0`, returns a pointer value at the +byte offset `%1` relative to `%0`. + +### Reference Counting + +These instructions handle reference counting of heap objects. Values of +strong reference type have ownership semantics for the referenced heap +object. Retain and release operations, however, are never implicit in +SIL and always must be explicitly performed where needed. Retains and +releases on the value may be freely moved, and balancing retains and +releases may deleted, so long as an owning retain count is maintained +for the uses of the value. + +All reference-counting operations are defined to work correctly on null +references (whether strong, unowned, or weak). A non-null reference must +actually refer to a valid object of the indicated type (or a subtype). +Address operands are required to be valid and non-null. + +While SIL makes reference-counting operations explicit, the SIL type +system also fully represents strength of reference. This is useful for +several reasons: + +1. Type-safety: it is impossible to erroneously emit SIL that naively + uses a `@weak` or `@unowned` reference as if it were a + strong reference. +2. Consistency: when a reference is kept in memory, instructions like + `copy_addr` and `destroy_addr` implicitly carry the right semantics + in the type of the address, rather than needing special variants + or flags. +3. Ease of tooling: SIL directly stores the user's intended strength of + reference, making it straightforward to generate instrumentation + that would convey this to a memory profiler. In principle, with only + a modest number of additions and restrictions on SIL, it would even + be possible to drop all reference-counting instructions and use the + type information to feed a garbage collector. + +#### strong\_retain + + sil-instruction ::= 'strong_retain' sil-operand + + strong_retain %0 : $T + // $T must be a reference type + +Increases the strong retain count of the heap object referenced by `%0`. + +#### strong\_retain\_autoreleased + + sil-instruction ::= 'strong_retain_autoreleased' sil-operand + + strong_retain_autoreleased %0 : $T + // $T must have a retainable pointer representation + +Retains the heap object referenced by `%0` using the Objective-C ARC +"autoreleased return value" optimization. The operand must be the result +of an `apply` instruction with an Objective-C method callee, and the +`strong_retain_autoreleased` instruction must be first use of the value +after the defining `apply` instruction. + +TODO: Specify all the other strong\_retain\_autoreleased constraints +here. + +#### strong\_release + + strong_release %0 : $T + // $T must be a reference type. + +Decrements the strong reference count of the heap object referenced by +`%0`. If the release operation brings the strong reference count of the +object to zero, the object is destroyed and `@weak` references are +cleared. When both its strong and unowned reference counts reach zero, +the object's memory is deallocated. + +#### strong\_retain\_unowned + + sil-instruction ::= 'strong_retain_unowned' sil-operand + + strong_retain_unowned %0 : $@unowned T + // $T must be a reference type + +Asserts that the strong reference count of the heap object referenced by +`%0` is still positive, then increases it by one. + +#### unowned\_retain + + sil-instruction ::= 'unowned_retain' sil-operand + + unowned_retain %0 : $@unowned T + // $T must be a reference type + +Increments the unowned reference count of the heap object underlying +`%0`. + +#### unowned\_release + + sil-instruction ::= 'unowned_release' sil-operand + + unowned_release %0 : $@unowned T + // $T must be a reference type + +Decrements the unowned reference count of the heap object referenced by +`%0`. When both its strong and unowned reference counts reach zero, the +object's memory is deallocated. + +#### load\_weak + + sil-instruction ::= 'load_weak' '[take]'? sil-operand + + load_weak [take] %0 : $*@sil_weak Optional + // $T must be an optional wrapping a reference type + +Increments the strong reference count of the heap object held in the +operand, which must be an initialized weak reference. The result is +value of type `$Optional`, except that it is `null` if the heap +object has begun deallocation. + +This operation must be atomic with respect to the final `strong_release` +on the operand heap object. It need not be atomic with respect to +`store_weak` operations on the same address. + +#### store\_weak + + sil-instruction ::= 'store_weak' sil-value 'to' '[initialization]'? sil-operand + + store_weak %0 to [initialization] %1 : $*@sil_weak Optional + // $T must be an optional wrapping a reference type + +Initializes or reassigns a weak reference. The operand may be `nil`. + +If `[initialization]` is given, the weak reference must currently either +be uninitialized or destroyed. If it is not given, the weak reference +must currently be initialized. + +This operation must be atomic with respect to the final `strong_release` +on the operand (source) heap object. It need not be atomic with respect +to `store_weak` or `load_weak` operations on the same address. + +#### fix\_lifetime + + sil-instruction :: 'fix_lifetime' sil-operand + + fix_lifetime %0 : $T + // Fix the lifetime of a value %0 + fix_lifetime %1 : $*T + // Fix the lifetime of the memory object referenced by %1 + +Acts as a use of a value operand, or of the value in memory referenced +by an address operand. Optimizations may not move operations that would +destroy the value, such as `release_value`, `strong_release`, +`copy_addr [take]`, or `destroy_addr`, past this instruction. + +#### mark\_dependence + + sil-instruction :: 'mark_dependence' sil-operand 'on' sil-operand + + %2 = mark_dependence %0 : $*T on %1 : $Builtin.NativeObject + +Indicates that the validity of the first operand depends on the value of +the second operand. Operations that would destroy the second value must +not be moved before any instructions which depend on the result of this +instruction, exactly as if the address had been obviously derived from +that operand (e.g. using `ref_element_addr`). + +The result is always equal to the first operand. The first operand will +typically be an address, but it could be an address in a non-obvious +form, such as a Builtin.RawPointer or a struct containing the same. +Transformations should be somewhat forgiving here. + +The second operand may have either object or address type. In the latter +case, the dependency is on the current value stored in the address. + +#### is\_unique + + sil-instruction ::= 'is_unique' sil-operand + + %1 = is_unique %0 : $*T + // $T must be a reference-counted type + // %1 will be of type Builtin.Int1 + +Checks whether %0 is the address of a unique reference to a memory +object. Returns 1 if the strong reference count is 1, and 0 if the +strong reference count is greater than 1. + +A discussion of the semantics can be found here: arcopts.is\_unique. + +#### is\_unique\_or\_pinned + + sil-instruction ::= 'is_unique_or_pinned' sil-operand + + %1 = is_unique_or_pinned %0 : $*T + // $T must be a reference-counted type + // %1 will be of type Builtin.Int1 + +Checks whether %0 is the address of either a unique reference to a +memory object or a reference to a pinned object. Returns 1 if the strong +reference count is 1 or the object has been marked pinned by +strong\_pin. + +#### copy\_block + + sil-instruction :: 'copy_block' sil-operand + + %1 = copy_block %0 : $@convention(block) T -> U + +Performs a copy of an Objective-C block. Unlike retains of other +reference-counted types, this can produce a different value from the +operand if the block is copied from the stack to the heap. + +### Literals + +These instructions bind SIL values to literal constants or to global +entities. + +#### function\_ref + + sil-instruction ::= 'function_ref' sil-function-name ':' sil-type + + %1 = function_ref @function : $@thin T -> U + // $@thin T -> U must be a thin function type + // %1 has type $T -> U + +Creates a reference to a SIL function. + +#### global\_addr + + sil-instruction ::= 'global_addr' sil-global-name ':' sil-type + + %1 = global_addr @foo : $*Builtin.Word + +Creates a reference to the address of a global variable. + +#### integer\_literal + + sil-instruction ::= 'integer_literal' sil-type ',' int-literal + + %1 = integer_literal $Builtin.Int, 123 + // $Builtin.Int must be a builtin integer type + // %1 has type $Builtin.Int + +Creates an integer literal value. The result will be of type +`Builtin.Int`, which must be a builtin integer type. The literal +value is specified using Swift's integer literal syntax. + +#### float\_literal + + sil-instruction ::= 'float_literal' sil-type ',' int-literal + + %1 = float_literal $Builtin.FP, 0x3F800000 + // $Builtin.FP must be a builtin floating-point type + // %1 has type $Builtin.FP + +Creates a floating-point literal value. The result will be of type +``Builtin.FP<n>`, which must be a builtin floating-point type. The literal value is specified as the bitwise representation of the floating point value, using Swift's hexadecimal integer literal syntax. string_literal``````` +: + + sil-instruction ::= 'string_literal' encoding string-literal + encoding ::= 'utf8' + encoding ::= 'utf16' + + %1 = string_literal "asdf" + // %1 has type $Builtin.RawPointer + +Creates a reference to a string in the global string table. The result +is a pointer to the data. The referenced string is always +null-terminated. The string literal value is specified using Swift's +string literal syntax (though `\()` interpolations are not allowed). + +### Dynamic Dispatch + +These instructions perform dynamic lookup of class and generic methods. +They share a common set of attributes: + + sil-method-attributes ::= '[' 'volatile'? ']' + +The `volatile` attribute on a dynamic dispatch instruction indicates +that the method lookup is semantically required (as, for example, in +Objective-C). When the type of a dynamic dispatch instruction's operand +is known, optimization passes can promote non-`volatile` dispatch +instructions into static `function_ref` instructions. + +If a dynamic dispatch instruction references an Objective-C method +(indicated by the `foreign` marker on a method reference, as in +`#NSObject.description!1.foreign`), then the instruction represents an +`objc_msgSend` invocation. `objc_msgSend` invocations can only be used +as the callee of an `apply` instruction or `partial_apply` instruction. +They cannot be stored or used as `apply` or `partial_apply` arguments. +`objc_msgSend` invocations must always be `volatile`. + +#### class\_method + + sil-instruction ::= 'class_method' sil-method-attributes? + sil-operand ',' sil-decl-ref ':' sil-type + + %1 = class_method %0 : $T, #T.method!1 : $@thin U -> V + // %0 must be of a class type or class metatype $T + // #T.method!1 must be a reference to a dynamically-dispatched method of T or + // of one of its superclasses, at uncurry level >= 1 + // %1 will be of type $U -> V + +Looks up a method based on the dynamic type of a class or class metatype +instance. It is undefined behavior if the class value is null and the +method is not an Objective-C method. + +If: + +- the instruction is not `[volatile]`, +- the referenced method is not a `foreign` method, +- and the static type of the class instance is known, or the method is + known to be final, + +then the instruction is a candidate for devirtualization optimization. A +devirtualization pass can consult the module's VTables\_ to find the SIL +function that implements the method and promote the instruction to a +static function\_ref\_. + +#### super\_method + + sil-instruction ::= 'super_method' sil-method-attributes? + sil-operand ',' sil-decl-ref ':' sil-type + + %1 = super_method %0 : $T, #Super.method!1.foreign : $@thin U -> V + // %0 must be of a non-root class type or class metatype $T + // #Super.method!1.foreign must be a reference to an ObjC method of T's + // superclass or of one of its ancestor classes, at uncurry level >= 1 + // %1 will be of type $@thin U -> V + +Looks up a method in the superclass of a class or class metatype +instance. Note that for native Swift methods, `super.method` calls are +statically dispatched, so this instruction is only valid for Objective-C +methods. It is undefined behavior if the class value is null and the +method is not an Objective-C method. + +#### witness\_method + + sil-instruction ::= 'witness_method' sil-method-attributes? + sil-type ',' sil-decl-ref ':' sil-type + + %1 = witness_method $T, #Proto.method!1 \ + : $@thin @cc(witness_method) U -> V + // $T must be an archetype + // #Proto.method!1 must be a reference to a method of one of the protocol + // constraints on T + // U -> V must be the type of the referenced method, + // generic on Self + // %1 will be of type $@thin U -> V + +Looks up the implementation of a protocol method for a generic type +variable constrained by that protocol. The result will be generic on the +`Self` archetype of the original protocol and have the `witness_method` +calling convention. If the referenced protocol is an `@objc` protocol, +the resulting type has the `objc` calling convention. + +#### dynamic\_method + + sil-instruction ::= 'dynamic_method' sil-method-attributes? + sil-operand ',' sil-decl-ref ':' sil-type + + %1 = dynamic_method %0 : $P, #X.method!1 : $@thin U -> V + // %0 must be of a protocol or protocol composition type $P, + // where $P contains the Swift.DynamicLookup protocol + // #X.method!1 must be a reference to an @objc method of any class + // or protocol type + // + // The "self" argument of the method type $@thin U -> V must be + // Builtin.ObjCPointer + +Looks up the implementation of an Objective-C method with the same +selector as the named method for the dynamic type of the value inside an +existential container. The "self" operand of the result function value +is represented using an opaque type, the value for which must be +projected out as a value of type `Builtin.ObjCPointer`. + +It is undefined behavior if the dynamic type of the operand does not +have an implementation for the Objective-C method with the selector to +which the `dynamic_method` instruction refers, or if that implementation +has parameter or result types that are incompatible with the method +referenced by `dynamic_method`. This instruction should only be used in +cases where its result will be immediately consumed by an operation that +performs the selector check itself (e.g., an `apply` that lowers to +`objc_msgSend`). To query whether the operand has an implementation for +the given method and safely handle the case where it does not, use +dynamic\_method\_br\_. + +### Function Application + +These instructions call functions or wrap them in partial application or +specialization thunks. + +#### apply + + sil-instruction ::= 'apply' '[nothrow]'? sil-value + sil-apply-substitution-list? + '(' (sil-value (',' sil-value)*)? ')' + ':' sil-type + + sil-apply-substitution-list ::= '<' sil-substitution + (',' sil-substitution)* '>' + sil-substitution ::= type '=' type + + %r = apply %0(%1, %2, ...) : $(A, B, ...) -> R + // Note that the type of the callee '%0' is specified *after* the arguments + // %0 must be of a concrete function type $(A, B, ...) -> R + // %1, %2, etc. must be of the argument types $A, $B, etc. + // %r will be of the return type $R + + %r = apply %0(%1, %2, ...) : $(T, U, ...) -> R + // %0 must be of a polymorphic function type $(T, U, ...) -> R + // %1, %2, etc. must be of the argument types after substitution $A, $B, etc. + // %r will be of the substituted return type $R' + +Transfers control to function `%0`, passing it the given arguments. In +the instruction syntax, the type of the callee is specified after the +argument list; the types of the argument and of the defined value are +derived from the function type of the callee. The input argument tuple +type is destructured, and each element is passed as an individual +argument. The `apply` instruction does no retaining or releasing of its +arguments by itself; the calling convention\_'s retain/release policy +must be handled by separate explicit `retain` and `release` +instructions. The return value will likewise not be implicitly retained +or released. + +The callee value must have function type. That function type may not +have an error result, except the instruction has the `nothrow` attribute +set. The `nothrow` attribute specifies that the callee has an error +result but does not actually throw. For the regular case of calling a +function with error result, use `try_apply`. + +NB: If the callee value is of a thick function type, `apply` currently +consumes the callee value at +1 strong retain count. + +If the callee is generic, all of its generic parameters must be bound by +the given substitution list. The arguments and return value is given +with these generic substitutions applied. + +#### partial\_apply + + sil-instruction ::= 'partial_apply' sil-value + sil-apply-substitution-list? + '(' (sil-value (',' sil-value)*)? ')' + ':' sil-type + + %c = partial_apply %0(%1, %2, ...) : $(Z..., A, B, ...) -> R + // Note that the type of the callee '%0' is specified *after* the arguments + // %0 must be of a concrete function type $(Z..., A, B, ...) -> R + // %1, %2, etc. must be of the argument types $A, $B, etc., + // of the tail part of the argument tuple of %0 + // %c will be of the partially-applied thick function type (Z...) -> R + + %c = partial_apply %0(%1, %2, ...) : $(Z..., T, U, ...) -> R + // %0 must be of a polymorphic function type $(T, U, ...) -> R + // %1, %2, etc. must be of the argument types after substitution $A, $B, etc. + // of the tail part of the argument tuple of %0 + // %r will be of the substituted thick function type $(Z'...) -> R' + +Creates a closure by partially applying the function `%0` to a partial +sequence of its arguments. In the instruction syntax, the type of the +callee is specified after the argument list; the types of the argument +and of the defined value are derived from the function type of the +callee. The closure context will be allocated with retain count 1 and +initialized to contain the values `%1`, `%2`, etc. The closed-over +values will not be retained; that must be done separately before the +`partial_apply`. The closure does however take ownership of the +partially applied arguments; when the closure reference count reaches +zero, the contained values will be destroyed. + +If the callee is generic, all of its generic parameters must be bound by +the given substitution list. The arguments are given with these generic +substitutions applied, and the resulting closure is of concrete function +type with the given substitutions applied. The generic parameters +themselves cannot be partially applied; all of them must be bound. The +result is always a concrete function. + +TODO: The instruction, when applied to a generic function, currently +implicitly performs abstraction difference transformations enabled by +the given substitutions, such as promoting address-only arguments and +returns to register arguments. This should be fixed. + +This instruction is used to implement both curry thunks and closures. A +curried function in Swift: + + func foo(a:A)(b:B)(c:C)(d:D) -> E { /* body of foo */ } + +emits curry thunks in SIL as follows (retains and releases omitted for +clarity): + + func @foo : $@thin A -> B -> C -> D -> E { + entry(%a : $A): + %foo_1 = function_ref @foo_1 : $@thin (B, A) -> C -> D -> E + %thunk = partial_apply %foo_1(%a) : $@thin (B, A) -> C -> D -> E + return %thunk : $B -> C -> D -> E + } + + func @foo_1 : $@thin (B, A) -> C -> D -> E { + entry(%b : $B, %a : $A): + %foo_2 = function_ref @foo_2 : $@thin (C, B, A) -> D -> E + %thunk = partial_apply %foo_2(%b, %a) : $@thin (C, B, A) -> D -> E + return %thunk : $(B, A) -> C -> D -> E + } + + func @foo_2 : $@thin (C, B, A) -> D -> E { + entry(%c : $C, %b : $B, %a : $A): + %foo_3 = function_ref @foo_3 : $@thin (D, C, B, A) -> E + %thunk = partial_apply %foo_3(%c, %b, %a) : $@thin (D, C, B, A) -> E + return %thunk : $(C, B, A) -> D -> E + } + + func @foo_3 : $@thin (D, C, B, A) -> E { + entry(%d : $D, %c : $C, %b : $B, %a : $A): + // ... body of foo ... + } + +A local function in Swift that captures context, such as `bar` in the +following example: + + func foo(x:Int) -> Int { + func bar(y:Int) -> Int { + return x + y + } + return bar(1) + } + +lowers to an uncurried entry point and is curried in the enclosing +function: + + func @bar : $@thin (Int, @box Int, *Int) -> Int { + entry(%y : $Int, %x_box : $@box Int, %x_address : $*Int): + // ... body of bar ... + } + + func @foo : $@thin Int -> Int { + entry(%x : $Int): + // Create a box for the 'x' variable + %x_box = alloc_box $Int + store %x to %x_box#1 : $*Int + + // Create the bar closure + %bar_uncurried = function_ref @bar : $(Int, Int) -> Int + %bar = partial_apply %bar_uncurried(%x_box#0, %x_box#1) \ + : $(Int, Builtin.ObjectPointer, *Int) -> Int + + // Apply it + %1 = integer_literal $Int, 1 + %ret = apply %bar(%1) : $(Int) -> Int + + // Clean up + release %bar : $(Int) -> Int + return %ret : $Int + } + +#### builtin + + sil-instruction ::= 'builtin' string-literal + sil-apply-substitution-list? + '(' (sil-operand (',' sil-operand)*)? ')' + ':' sil-type + + %1 = builtin "foo"(%1 : $T, %2 : $U) : $V + // "foo" must name a function in the Builtin module + +Invokes functionality built into the backend code generator, such as +LLVM-level instructions and intrinsics. + +### Metatypes + +These instructions access metatypes, either statically by type name or +dynamically by introspecting class or generic values. + +#### metatype + + sil-instruction ::= 'metatype' sil-type + + %1 = metatype $T.metatype + // %1 has type $T.metatype + +Creates a reference to the metatype object for type `T`. + +#### value\_metatype + + sil-instruction ::= 'value_metatype' sil-type ',' sil-operand + + %1 = value_metatype $T.metatype, %0 : $T + // %0 must be a value or address of type $T + // %1 will be of type $T.metatype + +Obtains a reference to the dynamic metatype of the value `%0`. + +#### existential\_metatype + + sil-instruction ::= 'existential_metatype' sil-type ',' sil-operand + + %1 = existential_metatype $P.metatype, %0 : $P + // %0 must be a value of class protocol or protocol composition + // type $P, or an address of address-only protocol type $*P + // %1 will be a $P.metatype value referencing the metatype of the + // concrete value inside %0 + +Obtains the metatype of the concrete value referenced by the existential +container referenced by `%0`. + +#### objc\_protocol + + sil-instruction ::= 'objc_protocol' protocol-decl : sil-type + + %0 = objc_protocol #ObjCProto : $Protocol + +*TODO* Fill this in. + +### Aggregate Types + +These instructions construct and project elements from structs, tuples, +and class instances. + +#### retain\_value + + sil-instruction ::= 'retain_value' sil-operand + + retain_value %0 : $A + +Retains a loadable value, which simply retains any references it holds. + +For trivial types, this is a no-op. For reference types, this is +equivalent to a `strong_retain`. For `@unowned` types, this is +equivalent to an `unowned_retain`. In each of these cases, those are the +preferred forms. + +For aggregate types, especially enums, it is typically both easier and +more efficient to reason about aggregate copies than it is to reason +about copies of the subobjects. + +#### release\_value + + sil-instruction ::= 'release_value' sil-operand + + release_value %0 : $A + +Destroys a loadable value, by releasing any retainable pointers within +it. + +This is defined to be equivalent to storing the operand into a stack +allocation and using 'destroy\_addr' to destroy the object there. + +For trivial types, this is a no-op. For reference types, this is +equivalent to a `strong_release`. For `@unowned` types, this is +equivalent to an `unowned_release`. In each of these cases, those are +the preferred forms. + +For aggregate types, especially enums, it is typically both easier and +more efficient to reason about aggregate destroys than it is to reason +about destroys of the subobjects. + +#### autorelease\_value + + sil-instruction ::= 'autorelease_value' sil-operand + + autorelease_value %0 : $A + +*TODO* Complete this section. + +#### tuple + + sil-instruction ::= 'tuple' sil-tuple-elements + sil-tuple-elements ::= '(' (sil-operand (',' sil-operand)*)? ')' + sil-tuple-elements ::= sil-type '(' (sil-value (',' sil-value)*)? ')' + + %1 = tuple (%a : $A, %b : $B, ...) + // $A, $B, etc. must be loadable non-address types + // %1 will be of the "simple" tuple type $(A, B, ...) + + %1 = tuple $(a:A, b:B, ...) (%a, %b, ...) + // (a:A, b:B, ...) must be a loadable tuple type + // %1 will be of the type $(a:A, b:B, ...) + +Creates a loadable tuple value by aggregating multiple loadable values. + +If the destination type is a "simple" tuple type, that is, it has no +keyword argument labels or variadic arguments, then the first notation +can be used, which interleaves the element values and types. If keyword +names or variadic fields are specified, then the second notation must be +used, which spells out the tuple type before the fields. + +#### tuple\_extract + + sil-instruction ::= 'tuple_extract' sil-operand ',' int-literal + + %1 = tuple_extract %0 : $(T...), 123 + // %0 must be of a loadable tuple type $(T...) + // %1 will be of the type of the selected element of %0 + +Extracts an element from a loadable tuple value. + +#### tuple\_element\_addr + + sil-instruction ::= 'tuple_element_addr' sil-operand ',' int-literal + + %1 = tuple_element_addr %0 : $*(T...), 123 + // %0 must of a $*(T...) address-of-tuple type + // %1 will be of address type $*U where U is the type of the 123rd + // element of T + +Given the address of a tuple in memory, derives the address of an +element within that value. + +#### struct + + sil-instruction ::= 'struct' sil-type '(' (sil-operand (',' sil-operand)*)? ')' + + %1 = struct $S (%a : $A, %b : $B, ...) + // $S must be a loadable struct type + // $A, $B, ... must be the types of the physical 'var' fields of $S in order + // %1 will be of type $S + +Creates a value of a loadable struct type by aggregating multiple +loadable values. + +#### struct\_extract + + sil-instruction ::= 'struct_extract' sil-operand ',' sil-decl-ref + + %1 = struct_extract %0 : $S, #S.field + // %0 must be of a loadable struct type $S + // #S.field must be a physical 'var' field of $S + // %1 will be of the type of the selected field of %0 + +Extracts a physical field from a loadable struct value. + +#### struct\_element\_addr + + sil-instruction ::= 'struct_element_addr' sil-operand ',' sil-decl-ref + + %1 = struct_element_addr %0 : $*S, #S.field + // %0 must be of a struct type $S + // #S.field must be a physical 'var' field of $S + // %1 will be the address of the selected field of %0 + +Given the address of a struct value in memory, derives the address of a +physical field within the value. + +#### ref\_element\_addr + + sil-instruction ::= 'ref_element_addr' sil-operand ',' sil-decl-ref + + %1 = ref_element_addr %0 : $C, #C.field + // %0 must be a value of class type $C + // #C.field must be a non-static physical field of $C + // %1 will be of type $*U where U is the type of the selected field + // of C + +Given an instance of a class, derives the address of a physical instance +variable inside the instance. It is undefined behavior if the class +value is null. + +### Enums + +These instructions construct values of enum type. Loadable enum values +are created with the enum\_ instruction. Address-only enums require +two-step initialization. First, if the case requires data, that data is +stored into the enum at the address projected by +init\_enum\_data\_addr\_. This step is skipped for cases without data. +Finally, the tag for the enum is injected with an inject\_enum\_addr\_ +instruction: + + enum AddressOnlyEnum { + case HasData(AddressOnlyType) + case NoData + } + + sil @init_with_data : $(AddressOnlyType) -> AddressOnlyEnum { + entry(%0 : $*AddressOnlyEnum, %1 : $*AddressOnlyType): + // Store the data argument for the case. + %2 = init_enum_data_addr %0 : $*AddressOnlyEnum, #AddressOnlyEnum.HasData + copy_addr [take] %2 to [initialization] %1 : $*AddressOnlyType + // Inject the tag. + inject_enum_addr %0 : $*AddressOnlyEnum, #AddressOnlyEnum.HasData + return + } + + sil @init_without_data : $() -> AddressOnlyEnum { + // No data. We only need to inject the tag. + inject_enum_addr %0 : $*AddressOnlyEnum, #AddressOnlyEnum.NoData + return + } + +Accessing the value of a loadable enum is inseparable from dispatching +on its discriminator and is done with the switch\_enum\_ terminator: + + enum Foo { case A(Int), B(String) } + + sil @switch_foo : $(Foo) -> () { + entry(%foo : $Foo): + switch_enum %foo : $Foo, case #Foo.A: a_dest, case #Foo.B: b_dest + + a_dest(%a : $Int): + /* use %a */ + + b_dest(%b : $String): + /* use %b */ + } + +An address-only enum can be tested by branching on it using the +switch\_enum\_addr\_ terminator. Its value can then be taken by +destructively projecting the enum value with +unchecked\_take\_enum\_data\_addr\_: + + enum Foo { case A(T), B(String) } + + sil @switch_foo : $ (Foo) -> () { + entry(%foo : $*Foo): + switch_enum_addr %foo : $*Foo, case #Foo.A: a_dest, case #Foo.B: b_dest + + a_dest: + %a = unchecked_take_enum_data_addr %foo : $*Foo, #Foo.A + /* use %a */ + + b_dest: + %b = unchecked_take_enum_data_addr %foo : $*Foo, #Foo.B + /* use %b */ + } + +#### enum + + sil-instruction ::= 'enum' sil-type ',' sil-decl-ref (',' sil-operand)? + + %1 = enum $U, #U.EmptyCase + %1 = enum $U, #U.DataCase, %0 : $T + // $U must be an enum type + // #U.DataCase or #U.EmptyCase must be a case of enum $U + // If #U.Case has a data type $T, %0 must be a value of type $T + // If #U.Case has no data type, the operand must be omitted + // %1 will be of type $U + +Creates a loadable enum value in the given `case`. If the `case` has a +data type, the enum value will contain the operand value. + +#### unchecked\_enum\_data + + sil-instruction ::= 'unchecked_enum_data' sil-operand ',' sil-decl-ref + + %1 = unchecked_enum_data %0 : $U, #U.DataCase + // $U must be an enum type + // #U.DataCase must be a case of enum $U with data + // %1 will be of object type $T for the data type of case U.DataCase + +Unsafely extracts the payload data for an enum `case` from an enum +value. It is undefined behavior if the enum does not contain a value of +the given case. + +#### init\_enum\_data\_addr + + sil-instruction ::= 'init_enum_data_addr' sil-operand ',' sil-decl-ref + + %1 = init_enum_data_addr %0 : $*U, #U.DataCase + // $U must be an enum type + // #U.DataCase must be a case of enum $U with data + // %1 will be of address type $*T for the data type of case U.DataCase + +Projects the address of the data for an enum `case` inside an enum. This +does not modify the enum or check its value. It is intended to be used +as part of the initialization sequence for an address-only enum. Storing +to the `init_enum_data_addr` for a case followed by `inject_enum_addr` +with that same case is guaranteed to result in a fully-initialized enum +value of that case being stored. Loading from the `init_enum_data_addr` +of an initialized enum value or injecting a mismatched case tag is +undefined behavior. + +The address is invalidated as soon as the operand enum is fully +initialized by an `inject_enum_addr`. + +#### inject\_enum\_addr + + sil-instruction ::= 'inject_enum_addr' sil-operand ',' sil-decl-ref + + inject_enum_addr %0 : $*U, #U.Case + // $U must be an enum type + // #U.Case must be a case of enum $U + // %0 will be overlaid with the tag for #U.Case + +Initializes the enum value referenced by the given address by overlaying +the tag for the given case. If the case has no data, this instruction is +sufficient to initialize the enum value. If the case has data, the data +must be stored into the enum at the `init_enum_data_addr` address for +the case *before* `inject_enum_addr` is applied. It is undefined +behavior if `inject_enum_addr` is applied for a case with data to an +uninitialized enum, or if `inject_enum_addr` is applied for a case with +data when data for a mismatched case has been stored to the enum. + +#### unchecked\_take\_enum\_data\_addr + + sil-instruction ::= 'unchecked_take_enum_data_addr' sil-operand ',' sil-decl-ref + + %1 = unchecked_take_enum_data_addr %0 : $*U, #U.DataCase + // $U must be an enum type + // #U.DataCase must be a case of enum $U with data + // %1 will be of address type $*T for the data type of case U.DataCase + +Invalidates an enum value, and takes the address of the payload for the +given enum `case` in-place in memory. The referenced enum value is no +longer valid, but the payload value referenced by the result address is +valid and must be destroyed. It is undefined behavior if the referenced +enum does not contain a value of the given `case`. The result shares +memory with the original enum value; the enum memory cannot be +reinitialized as an enum until the payload has also been invalidated. + +(1.0 only) + +For the first payloaded case of an enum, `unchecked_take_enum_data_addr` +is guaranteed to have no side effects; the enum value will not be +invalidated. + +#### select\_enum + + sil-instruction ::= 'select_enum' sil-operand sil-select-case* + (',' 'default' sil-value)? + ':' sil-type + + %n = select_enum %0 : $U, \ + case #U.Case1: %1, \ + case #U.Case2: %2, /* ... */ \ + default %3 : $T + + // $U must be an enum type + // #U.Case1, Case2, etc. must be cases of enum $U + // %1, %2, %3, etc. must have type $T + // %n has type $T + +Selects one of the "case" or "default" operands based on the case of an +enum value. This is equivalent to a trivial switch\_enum\_ branch +sequence: + + entry: + switch_enum %0 : $U, \ + case #U.Case1: bb1, \ + case #U.Case2: bb2, /* ... */ \ + default bb_default + bb1: + br cont(%1 : $T) // value for #U.Case1 + bb2: + br cont(%2 : $T) // value for #U.Case2 + bb_default: + br cont(%3 : $T) // value for default + cont(%n : $T): + // use argument %n + +but turns the control flow dependency into a data flow dependency. For +address-only enums, select\_enum\_addr\_ offers the same functionality +for an indirectly referenced enum value in memory. + +#### select\_enum\_addr + + sil-instruction ::= 'select_enum_addr' sil-operand sil-select-case* + (',' 'default' sil-value)? + ':' sil-type + + %n = select_enum_addr %0 : $*U, \ + case #U.Case1: %1, \ + case #U.Case2: %2, /* ... */ \ + default %3 : $T + + // %0 must be the address of an enum type $*U + // #U.Case1, Case2, etc. must be cases of enum $U + // %1, %2, %3, etc. must have type $T + // %n has type $T + +Selects one of the "case" or "default" operands based on the case of the +referenced enum value. This is the address-only counterpart to +select\_enum\_. + +### Protocol and Protocol Composition Types + +These instructions create and manipulate values of protocol and protocol +composition type. From SIL's perspective, protocol and protocol +composition types consist of an *existential container*, which is a +generic container for a value of unknown runtime type, referred to as an +"existential type" in type theory. The existential container consists of +a reference to the *witness table(s)* for the protocol(s) referred to by +the protocol type and a reference to the underlying *concrete value*, +which may be either stored in-line inside the existential container for +small values or allocated separately into a buffer owned and managed by +the existential container for larger values. + +Depending on the constraints applied to an existential type, an +existential container may use one of several representations: + +- **Opaque existential containers**: If none of the protocols in a + protocol type are class protocols, then the existential container + for that type is address-only and referred to in the implementation + as an *opaque existential container*. The value semantics of the + existential container propagate to the contained concrete value. + Applying `copy_addr` to an opaque existential container copies the + contained concrete value, deallocating or reallocating the + destination container's owned buffer if necessary. Applying + `destroy_addr` to an opaque existential container destroys the + concrete value and deallocates any buffers owned by the + existential container. The following instructions manipulate opaque + existential containers: + - init\_existential\_addr\_ + - open\_existential\_addr\_ + - deinit\_existential\_addr\_ +- **Class existential containers**: If a protocol type is constrained + by one or more class protocols, then the existential container for + that type is loadable and referred to in the implementation as a + *class existential container*. Class existential containers have + reference semantics and can be `retain`-ed and `release`-d. The + following instructions manipulate class existential containers: + - init\_existential\_ref\_ + - open\_existential\_ref\_ +- **Metatype existential containers**: Existential metatypes use a + container consisting of the type metadata for the conforming type + along with the protocol conformances. Metatype existential + containers are trivial types. The following instructions manipulate + metatype existential containers: + - init\_existential\_metatype\_ + - open\_existential\_metatype\_ +- **Boxed existential containers**: The standard library `ErrorType` + protocol uses a size-optimized reference-counted container, which + indirectly stores the conforming value. Boxed existential containers + can be `retain`-ed and `release`-d. The following instructions + manipulate boxed existential containers: + - alloc\_existential\_box\_ + - open\_existential\_box\_ + - dealloc\_existential\_box\_ + +Some existential types may additionally support specialized +representations when they contain certain known concrete types. For +example, when Objective-C interop is available, the `ErrorType` protocol +existential supports a class existential container representation for +`NSError` objects, so it can be initialized from one using +`init_existential_ref` instead of the more expensive +`alloc_existential_box`: + + bb(%nserror: $NSError): + // The slow general way to form an ErrorType, allocating a box and + // storing to its value buffer: + %error1 = alloc_existential_box $ErrorType, $NSError + strong_retain %nserror: $NSError + store %nserror to %error1#1 : $NSError + + // The fast path supported for NSError: + strong_retain %nserror: $NSError + %error2 = init_existential_ref %nserror: $NSError, $ErrorType + +#### init\_existential\_addr + + sil-instruction ::= 'init_existential_addr' sil-operand ',' sil-type + + %1 = init_existential_addr %0 : $*P, $T + // %0 must be of a $*P address type for non-class protocol or protocol + // composition type P + // $T must be an AST type that fulfills protocol(s) P + // %1 will be of type $*T', where T' is the maximally abstract lowering + // of type T + +Partially initializes the memory referenced by `%0` with an existential +container prepared to contain a value of type `$T`. The result of the +instruction is an address referencing the storage for the contained +value, which remains uninitialized. The contained value must be +`store`-d or `copy_addr`-ed to in order for the existential value to be +fully initialized. If the existential container needs to be destroyed +while the contained value is uninitialized, `deinit_existential_addr` +must be used to do so. A fully initialized existential container can be +destroyed with `destroy_addr` as usual. It is undefined behavior to +`destroy_addr` a partially-initialized existential container. + +#### deinit\_existential\_addr + + sil-instruction ::= 'deinit_existential_addr' sil-operand + + deinit_existential_addr %0 : $*P + // %0 must be of a $*P address type for non-class protocol or protocol + // composition type P + +Undoes the partial initialization performed by `init_existential_addr`. +`deinit_existential_addr` is only valid for existential containers that +have been partially initialized by `init_existential_addr` but haven't +had their contained value initialized. A fully initialized existential +must be destroyed with `destroy_addr`. + +#### open\_existential\_addr + + sil-instruction ::= 'open_existential_addr' sil-operand 'to' sil-type + + %1 = open_existential_addr %0 : $*P to $*@opened P + // %0 must be of a $*P type for non-class protocol or protocol composition + // type P + // $*@opened P must be a unique archetype that refers to an opened + // existential type P. + // %1 will be of type $*P + +Obtains the address of the concrete value inside the existential +container referenced by `%0`. The protocol conformances associated with +this existential container are associated directly with the archetype +`$*@opened P`. This pointer can be used with any operation on +archetypes, such as `witness_method`. + +#### init\_existential\_ref + + sil-instruction ::= 'init_existential_ref' sil-operand ':' sil-type ',' + sil-type + + %1 = init_existential_ref %0 : $C' : $C, $P + // %0 must be of class type $C', lowered from AST type $C, conforming to + // protocol(s) $P + // $P must be a class protocol or protocol composition type + // %1 will be of type $P + +Creates a class existential container of type `$P` containing a +reference to the class instance `%0`. + +#### open\_existential\_ref + + sil-instruction ::= 'open_existential_ref' sil-operand 'to' sil-type + + %1 = open_existential_ref %0 : $P to $@opened P + // %0 must be of a $P type for a class protocol or protocol composition + // $@opened P must be a unique archetype that refers to an opened + // existential type P + // %1 will be of type $@opened P + +Extracts the class instance reference from a class existential +container. The protocol conformances associated with this existential +container are associated directly with the archetype `@opened P`. This +pointer can be used with any operation on archetypes, such as +`witness_method`. When the operand is of metatype type, the result will +be the metatype of the opened archetype. + +#### init\_existential\_metatype + + sil-instruction ::= 'init_existential_metatype' sil-operand ',' sil-type + + %1 = init_existential_metatype $0 : $@ T.Type, $@ P.Type + // %0 must be of a metatype type $@ T.Type where T: P + // %@ P.Type must be the existential metatype of a protocol or protocol + // composition, with the same metatype representation + // %1 will be of type $@ P.Type + +Creates a metatype existential container of type `$P.Type` containing +the conforming metatype of `$T`. + +#### open\_existential\_metatype + + sil-instruction ::= 'open_existential_metatype' sil-operand 'to' sil-type + + %1 = open_existential_metatype %0 : $@ P.Type to $@ (@opened P).Type + // %0 must be of a $P.Type existential metatype for a protocol or protocol + // composition + // $@ (@opened P).Type must be the metatype of a unique archetype that + // refers to an opened existential type P, with the same metatype + // representation + // %1 will be of type $@ (@opened P).Type + +Extracts the metatype from an existential metatype. The protocol +conformances associated with this existential container are associated +directly with the archetype `@opened P`. + +#### alloc\_existential\_box + + sil-instruction ::= 'alloc_existential_box' sil-type ',' sil-type + + %1 = alloc_existential_box $P, $T + // $P must be a protocol or protocol composition type with boxed + // representation + // $T must be an AST type that conforms to P + // %1#0 will be of type $P + // %1#1 will be of type $*T', where T' is the most abstracted lowering of T + +Allocates a boxed existential container of type `$P` with space to hold +a value of type `$T'`. The box is not fully initialized until a valid +value has been stored into the box. If the box must be deallocated +before it is fully initialized, `dealloc_existential_box` must be used. +A fully initialized box can be `retain`-ed and `release`-d like any +reference-counted type. The address `%0#1` is dependent on the lifetime +of the owner reference `%0#0`. + +#### open\_existential\_box + + sil-instruction ::= 'open_existential_box' sil-operand 'to' sil-type + + %1 = open_existential_box %0 : $P to $*@opened P + // %0 must be a value of boxed protocol or protocol composition type $P + // %@opened P must be the address type of a unique archetype that refers to + /// an opened existential type P + // %1 will be of type $*@opened P + +Projects the address of the value inside a boxed existential container, +and uses the enclosed type and protocol conformance metadata to bind the +opened archetype `$@opened P`. The result address is dependent on both +the owning box and the enclosing function; in order to "open" a boxed +existential that has directly adopted a class reference, temporary +scratch space may need to have been allocated. + +#### dealloc\_existential\_box + + sil-instruction ::= 'dealloc_existential_box' sil-operand, sil-type + + dealloc_existential_box %0 : $P, $T + // %0 must be an uninitialized box of boxed existential container type $P + // $T must be the AST type for which the box was allocated + +Deallocates a boxed existential container. The value inside the +existential buffer is not destroyed; either the box must be +uninitialized, or the value must have been projected out and destroyed +beforehand. It is undefined behavior if the concrete type `$T` is not +the same type for which the box was allocated with +`alloc_existential_box`. + +### Blocks + +#### project\_block\_storage + + sil-instruction ::= 'project_block_storage' sil-operand ':' sil-type + +#### init\_block\_storage\_header + +*TODO* Fill this in. The printing of this instruction looks incomplete +on trunk currently. + +### Unchecked Conversions + +These instructions implement type conversions which are not checked. +These are either user-level conversions that are always safe and do not +need to be checked, or implementation detail conversions that are +unchecked for performance or flexibility. + +#### upcast + + sil-instruction ::= 'upcast' sil-operand 'to' sil-type + + %1 = upcast %0 : $D to $B + // $D and $B must be class types or metatypes, with B a superclass of D + // %1 will have type $B + +Represents a conversion from a derived class instance or metatype to a +superclass, or from a base-class-constrained archetype to its base +class. + +#### address\_to\_pointer + + sil-instruction ::= 'address_to_pointer' sil-operand 'to' sil-type + + %1 = address_to_pointer %0 : $*T to $Builtin.RawPointer + // %0 must be of an address type $*T + // %1 will be of type Builtin.RawPointer + +Creates a `Builtin.RawPointer` value corresponding to the address `%0`. +Converting the result pointer back to an address of the same type will +give an address equivalent to `%0`. It is undefined behavior to cast the +`RawPointer` to any address type other than its original address type or +any layout compatible types\_. + +#### pointer\_to\_address + + sil-instruction ::= 'pointer_to_address' sil-operand 'to' sil-type + + %1 = pointer_to_address %0 : $Builtin.RawPointer to $*T + // %1 will be of type $*T + +Creates an address value corresponding to the `Builtin.RawPointer` value +`%0`. Converting a `RawPointer` back to an address of the same type as +its originating `address_to_pointer` instruction gives back an +equivalent address. It is undefined behavior to cast the `RawPointer` +back to any type other than its original address type or +layout compatible types\_. It is also undefined behavior to cast a +`RawPointer` from a heap object to any address type. + +#### unchecked\_ref\_cast + + sil-instruction ::= 'unchecked_ref_cast' sil-operand 'to' sil-type + + %1 = unchecked_ref_cast %0 : $A to $B + // %0 must be an object of type $A + // $A must be a type with retainable pointer representation + // %1 will be of type $B + // $B must be a type with retainable pointer representation + +Converts a heap object reference to another heap object reference type. +This conversion is unchecked, and it is undefined behavior if the +destination type is not a valid type for the heap object. The heap +object reference on either side of the cast may be a class existential, +and may be wrapped in one level of Optional. + +#### unchecked\_ref\_cast\_addr + + sil-instruction ::= 'unchecked_ref_cast_addr' + sil-type 'in' sil-operand 'to' + sil-type 'in' sil-operand + + unchecked_ref_cast_addr $A in %0 : $*A to $B in %1 : $*B + // %0 must be the address of an object of type $A + // $A must be a type with retainable pointer representation + // %1 must be the address of storage for an object of type $B + // $B must be a retainable pointer representation + +Loads a heap object reference from an address and stores it at the +address of another uninitialized heap object reference. The loaded +reference is always taken, and the stored reference is initialized. This +conversion is unchecked, and it is undefined behavior if the destination +type is not a valid type for the heap object. The heap object reference +on either side of the cast may be a class existential, and may be +wrapped in one level of Optional. + +#### unchecked\_addr\_cast + + sil-instruction ::= 'unchecked_addr_cast' sil-operand 'to' sil-type + + %1 = unchecked_addr_cast %0 : $*A to $*B + // %0 must be an address + // %1 will be of type $*B + +Converts an address to a different address type. Using the resulting +address is undefined unless `B` is layout compatible with `A`. The +layout of `A` may be smaller than that of `B` as long as the lower order +bytes have identical layout. + +#### unchecked\_trivial\_bit\_cast + + sil-instruction ::= 'unchecked_trivial_bit_cast' sil-operand 'to' sil-type + + %1 = unchecked_trivial_bit_cast %0 : $Builtin.NativeObject to $Builtin.Word + // %0 must be an object. + // %1 must be an object with trivial type. + +Bitcasts an object of type `A` to be of same sized or smaller type `B` +with the constraint that `B` must be trivial. This can be used for +bitcasting among trivial types, but more importantly is a one way +bitcast from non-trivial types to trivial types. + +#### unchecked\_bitwise\_cast + + sil-instruction ::= 'unchecked_bitwise_cast' sil-operand 'to' sil-type + + %1 = unchecked_bitwise_cast %0 : $A to $B + +Bitwise copies an object of type `A` into a new object of type `B` of +the same size or smaller. + +#### ref\_to\_raw\_pointer + + sil-instruction ::= 'ref_to_raw_pointer' sil-operand 'to' sil-type + + %1 = ref_to_raw_pointer %0 : $C to $Builtin.RawPointer + // $C must be a class type, or Builtin.ObjectPointer, or Builtin.ObjCPointer + // %1 will be of type $Builtin.RawPointer + +Converts a heap object reference to a `Builtin.RawPointer`. The +`RawPointer` result can be cast back to the originating class type but +does not have ownership semantics. It is undefined behavior to cast a +`RawPointer` from a heap object reference to an address using +`pointer_to_address`. + +#### raw\_pointer\_to\_ref + + sil-instruction ::= 'raw_pointer_to_ref' sil-operand 'to' sil-type + + %1 = raw_pointer_to_ref %0 : $Builtin.RawPointer to $C + // $C must be a class type, or Builtin.ObjectPointer, or Builtin.ObjCPointer + // %1 will be of type $C + +Converts a `Builtin.RawPointer` back to a heap object reference. Casting +a heap object reference to `Builtin.RawPointer` back to the same type +gives an equivalent heap object reference (though the raw pointer has no +ownership semantics for the object on its own). It is undefined behavior +to cast a `RawPointer` to a type unrelated to the dynamic type of the +heap object. It is also undefined behavior to cast a `RawPointer` from +an address to any heap object type. + +#### ref\_to\_unowned + + sil-instruction ::= 'ref_to_unowned' sil-operand + + %1 = unowned_to_ref %0 : T + // $T must be a reference type + // %1 will have type $@unowned T + +Adds the `@unowned` qualifier to the type of a reference to a heap +object. No runtime effect. + +#### unowned\_to\_ref + + sil-instruction ::= 'unowned_to_ref' sil-operand + + %1 = unowned_to_ref %0 : $@unowned T + // $T must be a reference type + // %1 will have type $T + +Strips the `@unowned` qualifier off the type of a reference to a heap +object. No runtime effect. + +#### ref\_to\_unmanaged + +TODO + +#### unmanaged\_to\_ref + +TODO + +#### convert\_function + + sil-instruction ::= 'convert_function' sil-operand 'to' sil-type + + %1 = convert_function %0 : $T -> U to $T' -> U' + // %0 must be of a function type $T -> U ABI-compatible with $T' -> U' + // (see below) + // %1 will be of type $T' -> U' + +Performs a conversion of the function `%0` to type `T`, which must be +ABI-compatible with the type of `%0`. Function types are ABI-compatible +if their input and result types are tuple types that, after +destructuring, differ only in the following ways: + +- Corresponding tuple elements may add, remove, or change + keyword names. `(a:Int, b:Float, UnicodeScalar) -> ()` and + `(x:Int, Float, z:UnicodeScalar) -> ()` are ABI compatible. +- A class tuple element of the destination type may be a superclass or + subclass of the source type's corresponding tuple element. + +The function types may also differ in attributes, with the following +caveats: + +- The `convention` attribute cannot be changed. +- A `@noreturn` function may be converted to a non-`@noreturn` type + and vice-versa. + +#### thin\_function\_to\_pointer + +TODO + +#### pointer\_to\_thin\_function + +TODO + +#### ref\_to\_bridge\_object + + sil-instruction ::= 'ref_to_bridge_object' sil-operand, sil-operand + + %2 = ref_to_bridge_object %0 : $C, %1 : $Builtin.Word + // %1 must be of reference type $C + // %2 will be of type Builtin.BridgeObject + +Creates a `Builtin.BridgeObject` that references `%0`, with spare bits +in the pointer representation populated by bitwise-OR-ing in the value +of `%1`. It is undefined behavior if this bitwise OR operation affects +the reference identity of `%0`; in other words, after the following +instruction sequence: + + %b = ref_to_bridge_object %r : $C, %w : $Builtin.Word + %r2 = bridge_object_to_ref %b : $Builtin.BridgeObject to $C + +`%r` and `%r2` must be equivalent. In particular, it is assumed that +retaining or releasing the `BridgeObject` is equivalent to retaining or +releasing the original reference, and that the above +`ref_to_bridge_object` / `bridge_object_to_ref` round-trip can be folded +away to a no-op. + +On platforms with ObjC interop, there is additionally a +platform-specific bit in the pointer representation of a `BridgeObject` +that is reserved to indicate whether the referenced object has native +Swift refcounting. It is undefined behavior to set this bit when the +first operand references an Objective-C object. + +#### bridge\_object\_to\_ref + + sil-instruction ::= 'bridge_object_to_ref' sil-operand 'to' sil-type + + %1 = bridge_object_to_ref %0 : $Builtin.BridgeObject to $C + // $C must be a reference type + // %1 will be of type $C + +Extracts the object reference from a `Builtin.BridgeObject`, masking out +any spare bits. + +#### bridge\_object\_to\_word + + sil-instruction ::= 'bridge_object_to_word' sil-operand 'to' sil-type + + %1 = bridge_object_to_word %0 : $Builtin.BridgeObject to $Builtin.Word + // %1 will be of type $Builtin.Word + +Provides the bit pattern of a `Builtin.BridgeObject` as an integer. + +#### thin\_to\_thick\_function + + sil-instruction ::= 'thin_to_thick_function' sil-operand 'to' sil-type + + %1 = thin_to_thick_function %0 : $@convention(thin) T -> U to $T -> U + // %0 must be of a thin function type $@convention(thin) T -> U + // The destination type must be the corresponding thick function type + // %1 will be of type $T -> U + +Converts a thin function value, that is, a bare function pointer with no +context information, into a thick function value with ignored context. +Applying the resulting thick function value is equivalent to applying +the original thin value. The `thin_to_thick_function` conversion may be +eliminated if the context is proven not to be needed. + +#### thick\_to\_objc\_metatype + + sil-instruction ::= 'thick_to_objc_metatype' sil-operand 'to' sil-type + + %1 = thick_to_objc_metatype %0 : $@thick T.metatype to $@objc_metatype T.metatype + // %0 must be of a thick metatype type $@thick T.metatype + // The destination type must be the corresponding Objective-C metatype type + // %1 will be of type $@objc_metatype T.metatype + +Converts a thick metatype to an Objective-C class metatype. `T` must be +of class, class protocol, or class protocol composition type. + +#### objc\_to\_thick\_metatype + + sil-instruction ::= 'objc_to_thick_metatype' sil-operand 'to' sil-type + + %1 = objc_to_thick_metatype %0 : $@objc_metatype T.metatype to $@thick T.metatype + // %0 must be of an Objective-C metatype type $@objc_metatype T.metatype + // The destination type must be the corresponding thick metatype type + // %1 will be of type $@thick T.metatype + +Converts an Objective-C class metatype to a thick metatype. `T` must be +of class, class protocol, or class protocol composition type. + +#### objc\_metatype\_to\_object + +TODO + +#### objc\_existential\_metatype\_to\_object + +TODO + +#### is\_nonnull + + sil-instruction ::= 'is_nonnull' sil-operand + + %1 = is_nonnull %0 : $C + // %0 must be of reference or function type $C + // %1 will be of type Builtin.Int1 + +Checks whether a reference type value is null, returning 1 if the value +is not null, or 0 if it is null. If the value is a function type, it +checks the function pointer (not the data pointer) for null. + +This is not a sensical thing for SIL to represent given that reference +types are non-nullable, but makes sense at the machine level. This is a +horrible hack that should go away someday. + +### Checked Conversions + +Some user-level cast operations can fail and thus require runtime +checking. + +The unconditional\_checked\_cast\_addr\_ and +unconditional\_checked\_cast\_ instructions performs an unconditional +checked cast; it is a runtime failure if the cast fails. The +checked\_cast\_addr\_br\_ and checked\_cast\_br\_ terminator instruction +performs a conditional checked cast; it branches to one of two +destinations based on whether the cast succeeds or not. + +#### unconditional\_checked\_cast + + sil-instruction ::= 'unconditional_checked_cast' sil-operand 'to' sil-type + + %1 = unconditional_checked_cast %0 : $A to $B + %1 = unconditional_checked_cast %0 : $*A to $*B + // $A and $B must be both objects or both addresses + // %1 will be of type $B or $*B + +Performs a checked scalar conversion, causing a runtime failure if the +conversion fails. + +#### unconditional\_checked\_cast\_addr + + sil-instruction ::= 'unconditional_checked_cast_addr' + sil-cast-consumption-kind + sil-type 'in' sil-operand 'to' + sil-type 'in' sil-operand + sil-cast-consumption-kind ::= 'take_always' + sil-cast-consumption-kind ::= 'take_on_success' + sil-cast-consumption-kind ::= 'copy_on_success' + + %1 = unconditional_checked_cast_addr take_on_success $A in %0 : $*@thick A to $B in $*@thick B + // $A and $B must be both addresses + // %1 will be of type $*B + +Performs a checked indirect conversion, causing a runtime failure if the +conversion fails. + +### Runtime Failures + +#### cond\_fail + + sil-instruction ::= 'cond_fail' sil-operand + + cond_fail %0 : $Builtin.Int1 + // %0 must be of type $Builtin.Int1 + +This instruction produces a runtime failure\_ if the operand is one. +Execution proceeds normally if the operand is zero. + +### Terminators + +These instructions terminate a basic block. Every basic block must end +with a terminator. Terminators may only appear as the final instruction +of a basic block. + +#### unreachable + + sil-terminator ::= 'unreachable' + + unreachable + +Indicates that control flow must not reach the end of the current basic +block. It is a dataflow error if an unreachable terminator is reachable +from the entry point of a function and is not immediately preceded by an +`apply` of a `@noreturn` function. + +#### return + + sil-terminator ::= 'return' sil-operand + + return %0 : $T + // $T must be the return type of the current function + +Exits the current function and returns control to the calling function. +If the current function was invoked with an `apply` instruction, the +result of that function will be the operand of this `return` +instruction. If the current function was invoked with a +`` try_apply` instruction, control resumes at the normal destination, and the value of the basic block argument will be the operand of this ``return`instruction.`return`does not retain or release its operand or any other values. A function must not contain more than one`return`instruction. autorelease_return````````` +: + + sil-terminator ::= 'autorelease_return' sil-operand + + autorelease_return %0 : $T + // $T must be the return type of the current function, which must be of + // class type + +Exits the current function and returns control to the calling function. +The result of the `apply` instruction that invoked the current function +will be the operand of this `return` instruction. The return value is +autoreleased into the active Objective-C autorelease pool using the +"autoreleased return value" optimization. The current function must use +the `@cc(objc_method)` calling convention. + +#### throw + + sil-terminator ::= 'throw' sil-operand + + throw %0 : $T + // $T must be the error result type of the current function + +Exits the current function and returns control to the calling function. +The current function must have an error result, and so the function must +have been invoked with a +`` try_apply` instruction. Control will resume in the error destination of that instruction, and the basic block argument will be the operand of the ``throw`.`throw`does not retain or release its operand or any other values. A function must not contain more than one`throw`instruction. br` +: + + sil-terminator ::= 'br' sil-identifier + '(' (sil-operand (',' sil-operand)*)? ')' + + br label (%0 : $A, %1 : $B, ...) + // `label` must refer to a basic block label within the current function + // %0, %1, etc. must be of the types of `label`'s arguments + +Unconditionally transfers control from the current basic block to the +block labeled `label`, binding the given values to the arguments of the +destination basic block. + +#### cond\_br + + sil-terminator ::= 'cond_br' sil-operand ',' + sil-identifier '(' (sil-operand (',' sil-operand)*)? ')' ',' + sil-identifier '(' (sil-operand (',' sil-operand)*)? ')' + + cond_br %0 : $Builtin.Int1, true_label (%a : $A, %b : $B, ...), \ + false_label (%x : $X, %y : $Y, ...) + // %0 must be of $Builtin.Int1 type + // `true_label` and `false_label` must refer to block labels within the + // current function and must not be identical + // %a, %b, etc. must be of the types of `true_label`'s arguments + // %x, %y, etc. must be of the types of `false_label`'s arguments + +Conditionally branches to `true_label` if `%0` is equal to `1` or to +`false_label` if `%0` is equal to `0`, binding the corresponding set of +values to the arguments of the chosen destination block. + +#### switch\_value + + sil-terminator ::= 'switch_value' sil-operand + (',' sil-switch-value-case)* + (',' sil-switch-default)? + sil-switch-value-case ::= 'case' sil-value ':' sil-identifier + sil-switch-default ::= 'default' sil-identifier + + switch_value %0 : $Builtin.Int, case %1: label1, \ + case %2: label2, \ + ..., \ + default labelN + + // %0 must be a value of builtin integer type $Builtin.Int + // `label1` through `labelN` must refer to block labels within the current + // function + // FIXME: All destination labels currently must take no arguments + +Conditionally branches to one of several destination basic blocks based +on a value of builtin integer or function type. If the operand value +matches one of the `case` values of the instruction, control is +transferred to the corresponding basic block. If there is a `default` +basic block, control is transferred to it if the value does not match +any of the `case` values. It is undefined behavior if the value does not +match any cases and no `default` branch is provided. + +#### select\_value + + sil-instruction ::= 'select_value' sil-operand sil-select-value-case* + (',' 'default' sil-value)? + ':' sil-type + sil-selct-value-case ::= 'case' sil-value ':' sil-value + + + %n = select_value %0 : $U, \ + case %c1: %r1, \ + case %c2: %r2, /* ... */ \ + default %r3 : $T + + // $U must be a builtin type. Only integers types are supported currently. + // c1, c2, etc must be of type $U + // %r1, %r2, %r3, etc. must have type $T + // %n has type $T + +Selects one of the "case" or "default" operands based on the case of an +value. This is equivalent to a trivial switch\_value\_ branch sequence: + + entry: + switch_value %0 : $U, \ + case %c1: bb1, \ + case %c2: bb2, /* ... */ \ + default bb_default + bb1: + br cont(%r1 : $T) // value for %c1 + bb2: + br cont(%r2 : $T) // value for %c2 + bb_default: + br cont(%r3 : $T) // value for default + cont(%n : $T): + // use argument %n + +but turns the control flow dependency into a data flow dependency. + +#### switch\_enum + + sil-terminator ::= 'switch_enum' sil-operand + (',' sil-switch-enum-case)* + (',' sil-switch-default)? + sil-switch-enum-case ::= 'case' sil-decl-ref ':' sil-identifier + + switch_enum %0 : $U, case #U.Foo: label1, \ + case #U.Bar: label2, \ + ..., \ + default labelN + + // %0 must be a value of enum type $U + // #U.Foo, #U.Bar, etc. must be 'case' declarations inside $U + // `label1` through `labelN` must refer to block labels within the current + // function + // label1 must take either no basic block arguments, or a single argument + // of the type of #U.Foo's data + // label2 must take either no basic block arguments, or a single argument + // of the type of #U.Bar's data, etc. + // labelN must take no basic block arguments + +Conditionally branches to one of several destination basic blocks based +on the discriminator in a loadable `enum` value. Unlike `switch_int`, +`switch_enum` requires coverage of the operand type: If the `enum` type +is resilient, the `default` branch is required; if the `enum` type is +fragile, the `default` branch is required unless a destination is +assigned to every `case` of the `enum`. The destination basic block for +a `case` may take an argument of the corresponding `enum` `case`'s data +type (or of the address type, if the operand is an address). If the +branch is taken, the destination's argument will be bound to the +associated data inside the original enum value. For example: + + enum Foo { + case Nothing + case OneInt(Int) + case TwoInts(Int, Int) + } + + sil @sum_of_foo : $Foo -> Int { + entry(%x : $Foo): + switch_enum %x : $Foo, \ + case #Foo.Nothing: nothing, \ + case #Foo.OneInt: one_int, \ + case #Foo.TwoInts: two_ints + + nothing: + %zero = integer_literal 0 : $Int + return %zero : $Int + + one_int(%y : $Int): + return %y : $Int + + two_ints(%ab : $(Int, Int)): + %a = tuple_extract %ab : $(Int, Int), 0 + %b = tuple_extract %ab : $(Int, Int), 1 + %add = function_ref @add : $(Int, Int) -> Int + %result = apply %add(%a, %b) : $(Int, Int) -> Int + return %result : $Int + } + +On a path dominated by a destination block of `switch_enum`, copying or +destroying the basic block argument has equivalent reference counting +semantics to copying or destroying the `switch_enum` operand: + + // This retain_value... + retain_value %e1 : $Enum + switch_enum %e1, case #Enum.A: a, case #Enum.B: b + +> a(%a : \$A): +> +> : // ...is balanced by this release\_value release\_value %a +> +> b(%b : \$B): +> +> : // ...and this one release\_value %b +> +#### switch\_enum\_addr + + sil-terminator ::= 'switch_enum_addr' sil-operand + (',' sil-switch-enum-case)* + (',' sil-switch-default)? + + switch_enum_addr %0 : $*U, case #U.Foo: label1, \ + case #U.Bar: label2, \ + ..., \ + default labelN + + // %0 must be the address of an enum type $*U + // #U.Foo, #U.Bar, etc. must be cases of $U + // `label1` through `labelN` must refer to block labels within the current + // function + // The destinations must take no basic block arguments + +Conditionally branches to one of several destination basic blocks based +on the discriminator in the enum value referenced by the address +operand. + +Unlike `switch_int`, `switch_enum` requires coverage of the operand +type: If the `enum` type is resilient, the `default` branch is required; +if the `enum` type is fragile, the `default` branch is required unless a +destination is assigned to every `case` of the `enum`. Unlike +`switch_enum`, the payload value is not passed to the destination basic +blocks; it must be projected out separately with +unchecked\_take\_enum\_data\_addr\_. + +#### dynamic\_method\_br + + sil-terminator ::= 'dynamic_method_br' sil-operand ',' sil-decl-ref + ',' sil-identifier ',' sil-identifier + + dynamic_method_br %0 : $P, #X.method!1, bb1, bb2 + // %0 must be of protocol type + // #X.method!1 must be a reference to an @objc method of any class + // or protocol type + +Looks up the implementation of an Objective-C method with the same +selector as the named method for the dynamic type of the value inside an +existential container. The "self" operand of the result function value +is represented using an opaque type, the value for which must be +projected out as a value of type `Builtin.ObjCPointer`. + +If the operand is determined to have the named method, this instruction +branches to `bb1`, passing it the uncurried function corresponding to +the method found. If the operand does not have the named method, this +instruction branches to `bb2`. + +#### checked\_cast\_br + + sil-terminator ::= 'checked_cast_br' sil-checked-cast-exact? + sil-operand 'to' sil-type ',' + sil-identifier ',' sil-identifier + sil-checked-cast-exact ::= '[' 'exact' ']' + + checked_cast_br %0 : $A to $B, bb1, bb2 + checked_cast_br %0 : $*A to $*B, bb1, bb2 + checked_cast_br [exact] %0 : $A to $A, bb1, bb2 + // $A and $B must be both object types or both address types + // bb1 must take a single argument of type $B or $*B + // bb2 must take no arguments + +Performs a checked scalar conversion from `$A` to `$B`. If the +conversion succeeds, control is transferred to `bb1`, and the result of +the cast is passed into `bb1` as an argument. If the conversion fails, +control is transferred to `bb2`. + +An exact cast checks whether the dynamic type is exactly the target +type, not any possible subtype of it. The source and target types must +be class types. + +#### checked\_cast\_addr\_br + + sil-terminator ::= 'checked_cast_addr_br' + sil-cast-consumption-kind + sil-type 'in' sil-operand 'to' + sil-stype 'in' sil-operand ',' + sil-identifier ',' sil-identifier + sil-cast-consumption-kind ::= 'take_always' + sil-cast-consumption-kind ::= 'take_on_success' + sil-cast-consumption-kind ::= 'copy_on_success' + + checked_cast_addr_br take_always $A in %0 : $*@thick A to $B in %2 : $*@thick B, bb1, bb2 + // $A and $B must be both address types + // bb1 must take a single argument of type $*B + // bb2 must take no arguments + +Performs a checked indirect conversion from `$A` to `$B`. If the +conversion succeeds, control is transferred to `bb1`, and the result of +the cast is left in the destination. If the conversion fails, control is +transferred to `bb2`. + +#### try\_apply + + sil-terminator ::= 'try_apply' sil-value + sil-apply-substitution-list? + '(' (sil-value (',' sil-value)*)? ')' + ':' sil-type + 'normal' sil-identifier, 'error' sil-identifier + + try_apply %0(%1, %2, ...) : $(A, B, ...) -> (R, @error E), + normal bb1, error bb2 + bb1(%3 : R): + bb2(%4 : E): + + // Note that the type of the callee '%0' is specified *after* the arguments + // %0 must be of a concrete function type $(A, B, ...) -> (R, @error E) + // %1, %2, etc. must be of the argument types $A, $B, etc. + +Transfers control to the function specified by `%0`, passing it the +given arguments. When `%0` returns, control resumes in either the normal +destination (if it returns with `return`) or the error destination (if +it returns with `throw`). + +`%0` must have a function type with an error result. + +The rules on generic substitutions are identical to those of `apply`. + +### Assertion configuration + +To be able to support disabling assertions at compile time there is a +builtin `assertion_configuration` function. A call to this function can +be replaced at compile time by a constant or can stay opaque. + +All calls to the `assert_configuration` function are replaced by the +constant propagation pass to the appropriate constant depending on +compile time settings. Subsequent passes remove dependent unwanted +control flow. Using this mechanism we support conditionally +enabling/disabling of code in SIL libraries depending on the assertion +configuration selected when the library is linked into user code. + +There are three assertion configurations: Debug (0), Release (1) and +DisableReplacement (-1). + +The optimization flag or a special assert configuration flag determines +the value. Depending on the configuration value assertions in the +standard library will be executed or not. + +The standard library uses this builtin to define an assert that can be +disabled at compile time. + + func assert(...) { + if (Int32(Builtin.assert_configuration()) == 0) { + _fatal_error_message(message, ...) + } + } + +The `assert_configuration` function application is serialized when we +build the standard library (we recognize the `-parse-stdlib` option and +don't do the constant replacement but leave the function application to +be serialized to sil). + +The compiler flag that influences the value of the +`assert_configuration` funtion application is the optimization flag: at +`` -Onone` the application will be replaced by ``Debug`at higher optimization levels the instruction will be replaced by`Release`. Optionally, the value to use for replacement can be specified with the`-AssertConf`flag which overwrites the value selected by the optimization flag (possible values are`Debug`,`Release`,`DisableReplacement`). If the call to the`assert\_configuration\`\` +function stays opaque until IRGen, IRGen will replace the application by +the constant representing Debug mode (0). This happens we can build the +standard library .dylib. The generate sil will retain the function call +but the generated .dylib will contain code with assertions enabled. diff --git a/docs/SIL.rst b/docs/SIL.rst deleted file mode 100644 index 208c7146bb4c7..0000000000000 --- a/docs/SIL.rst +++ /dev/null @@ -1,4330 +0,0 @@ -.. @raise litre.TestsAreMissing - -Swift Intermediate Language (SIL) -================================= - -.. contents:: - -Abstract --------- - -SIL is an SSA-form IR with high-level semantic information designed to implement -the Swift programming language. SIL accommodates the following use cases: - -- A set of guaranteed high-level optimizations that provide a predictable - baseline for runtime and diagnostic behavior. -- Diagnostic dataflow analysis passes that enforce Swift language requirements, - such as definitive initialization of variables and constructors, code - reachability, switch coverage. -- High-level optimization passes, including retain/release optimization, - dynamic method devirtualization, closure inlining, memory allocation promotion, - and generic function instantiation. -- A stable distribution format that can be used to distribute "fragile" - inlineable or generic code with Swift library modules, to be optimized into - client binaries. - -In contrast to LLVM IR, SIL is a generally target-independent format -representation that can be used for code distribution, but it can also express -target-specific concepts as well as LLVM can. - -SIL in the Swift Compiler -------------------------- - -At a high level, the Swift compiler follows a strict pipeline architecture: - -- The *Parse* module constructs an AST from Swift source code. -- The *Sema* module type-checks the AST and annotates it with type information. -- The *SILGen* module generates *raw SIL* from an AST. -- A series of *Guaranteed Optimization Passes* and *Diagnostic Passes* are run - over the raw SIL both to perform optimizations and to emit - language-specific diagnostics. These are always run, even at -Onone, and - produce *canonical SIL*. -- General SIL *Optimization Passes* optionally run over the canonical SIL to - improve performance of the resulting executable. These are enabled and - controlled by the optimization level and are not run at -Onone. -- *IRGen* lowers canonical SIL to LLVM IR. -- The LLVM backend (optionally) applies LLVM optimizations, runs the LLVM code - generator and emits binary code. - -The stages pertaining to SIL processing in particular are as follows: - -SILGen -~~~~~~ - -SILGen produces *raw SIL* by walking a type-checked Swift AST. -The form of SIL emitted by SILGen has the following properties: - -- Variables are represented by loading and storing mutable memory locations - instead of being in strict SSA form. This is similar to the initial - ``alloca``-heavy LLVM IR emitted by frontends such as Clang. However, Swift - represents variables as reference-counted "boxes" in the most general case, - which can be retained, released, and captured into closures. -- Dataflow requirements, such as definitive assignment, function returns, - switch coverage (TBD), etc. have not yet been enforced. -- ``transparent`` function optimization has not yet been honored. - -These properties are addressed by subsequent guaranteed optimization and -diagnostic passes which are always run against the raw SIL. - -Guaranteed Optimization and Diagnostic Passes -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -After SILGen, a deterministic sequence of optimization passes is run over the -raw SIL. We do not want the diagnostics produced by the compiler to change as -the compiler evolves, so these passes are intended to be simple and -predictable. - -- **Mandatory inlining** inlines calls to "transparent" functions. -- **Memory promotion** is implemented as two optimization phases, the first - of which performs capture analysis to promote ``alloc_box`` instructions to - ``alloc_stack``, and the second of which promotes non-address-exposed ``alloc_stack`` - instructions to SSA registers. -- **Constant propagation** folds constant expressions and propagates the constant values. - If an arithmetic overflow occurs during the constant expression computation, a diagnostic - is issued. -- **Return analysis** verifies that each function returns a value on every - code path and doesn't "fall of the end" of its definition, which is an error. - It also issues an error when a ``noreturn`` function returns. -- **Critical edge splitting** splits all critical edges from terminators that - don't support arbitrary basic block arguments (all non cond_branch - terminators). - -If all diagnostic passes succeed, the final result is the -*canonical SIL* for the program. - -TODO: - -- Generic specialization -- Basic ARC optimization for acceptable performance at -Onone. - -General Optimization Passes -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -SIL captures language-specific type information, making it possible to -perform high-level optimizations that are difficult to perform on LLVM -IR. - -- **Generic Specialization** analyzes specialized calls to generic - functions and generates new specialized version of the - functions. Then it rewrites all specialized usages of the generic - to a direct call of the appropriate specialized function. -- **Witness and VTable Devirtualization** for a given type looks up - the associated method from a class's vtable or a types witness table - and replaces the indirect virtual call with a call to the mapped - function. -- **Performance Inlining** -- **Reference Counting Optimizations** -- **Memory Promotion/Optimizations** -- **High-level domain specific optimizations** The swift compiler implements - high-level optimizations on basic Swift containers such as Array or String. - Domain specific optimizations require a defined interface between - the standard library and the optimizer. More details can be found here: - :ref:`HighLevelSILOptimizations` - -Syntax ------- - -SIL is reliant on Swift's type system and declarations, so SIL syntax -is an extension of Swift's. A ``.sil`` file is a Swift source file -with added SIL definitions. The Swift source is parsed only for its -declarations; Swift ``func`` bodies (except for nested declarations) -and top-level code are ignored by the SIL parser. In a ``.sil`` file, -there are no implicit imports; the ``swift`` and/or ``Builtin`` -standard modules must be imported explicitly if used. - -Here is an example of a ``.sil`` file:: - - sil_stage canonical - - import Swift - - // Define types used by the SIL function. - - struct Point { - var x : Double - var y : Double - } - - class Button { - func onClick() - func onMouseDown() - func onMouseUp() - } - - // Declare a Swift function. The body is ignored by SIL. - func taxicabNorm(a:Point) -> Double { - return a.x + a.y - } - - // Define a SIL function. - // The name @_T5norms11taxicabNormfT1aV5norms5Point_Sd is the mangled name - // of the taxicabNorm Swift function. - sil @_T5norms11taxicabNormfT1aV5norms5Point_Sd : $(Point) -> Double { - bb0(%0 : $Point): - // func Swift.+(Double, Double) -> Double - %1 = function_ref @_Tsoi1pfTSdSd_Sd - %2 = struct_extract %0 : $Point, #Point.x - %3 = struct_extract %0 : $Point, #Point.y - %4 = apply %1(%2, %3) : $(Double, Double) -> Double - %5 = return %4 : Double - } - - // Define a SIL vtable. This matches dynamically-dispatched method - // identifiers to their implementations for a known static class type. - sil_vtable Button { - #Button.onClick!1: @_TC5norms6Button7onClickfS0_FT_T_ - #Button.onMouseDown!1: @_TC5norms6Button11onMouseDownfS0_FT_T_ - #Button.onMouseUp!1: @_TC5norms6Button9onMouseUpfS0_FT_T_ - } - -SIL Stage -~~~~~~~~~ -:: - - decl ::= sil-stage-decl - sil-stage-decl ::= 'sil_stage' sil-stage - - sil-stage ::= 'raw' - sil-stage ::= 'canonical' - -There are different invariants on SIL depending on what stage of processing -has been applied to it. - -* **Raw SIL** is the form produced by SILGen that has not been run through - guaranteed optimizations or diagnostic passes. Raw SIL may not have a - fully-constructed SSA graph. It may contain dataflow errors. Some instructions - may be represented in non-canonical forms, such as ``assign`` and - ``destroy_addr`` for non-address-only values. Raw SIL should not be used - for native code generation or distribution. - -* **Canonical SIL** is SIL as it exists after guaranteed optimizations and - diagnostics. Dataflow errors must be eliminated, and certain instructions - must be canonicalized to simpler forms. Performance optimization and native - code generation are derived from this form, and a module can be distributed - containing SIL in this (or later) forms. - -SIL files declare the processing stage of the included SIL with one of the -declarations ``sil_stage raw`` or ``sil_stage canonical`` at top level. Only -one such declaration may appear in a file. - -SIL Types -~~~~~~~~~ -:: - - sil-type ::= '$' '*'? generic-parameter-list? type - -SIL types are introduced with the ``$`` sigil. SIL's type system is -closely related to Swift's, and so the type after the ``$`` is parsed -largely according to Swift's type grammar. - -Type Lowering -````````````` - -A *formal type* is the type of a value in Swift, such as an expression -result. Swift's formal type system intentionally abstracts over a -large number of representational issues like ownership transfer -conventions and directness of arguments. However, SIL aims to -represent most such implementation details, and so these differences -deserve to be reflected in the SIL type system. *Type lowering* is -the process of turning a formal type into its *lowered type*. - -It is important to be aware that the lowered type of a declaration -need not be the lowered type of the formal type of that declaration. -For example, the lowered type of a declaration reference: - -- will usually be thin, - -- will frequently be uncurried, - -- may have a non-Swift calling convention, - -- may use bridged types in its interface, and - -- may use ownership conventions that differ from Swift's default - conventions. - -Abstraction Difference -`````````````````````` - -Generic functions working with values of unconstrained type must -generally work with them indirectly, e.g. by allocating sufficient -memory for them and then passing around pointers to that memory. -Consider a generic function like this: - -:: - - func generateArray(n : Int, generator : () -> T) -> T[] - -The function ``generator`` will be expected to store its result -indirectly into an address passed in an implicit parameter. There's -really just no reasonable alternative when working with a value of -arbitrary type: - -- We don't want to generate a different copy of ``generateArray`` for - every type ``T``. - -- We don't want to give every type in the language a common - representation. - -- We don't want to dynamically construct a call to ``generator`` - depending on the type ``T``. - -But we also don't want the existence of the generic system to force -inefficiencies on non-generic code. For example, we'd like a function -of type ``() -> Int`` to be able to return its result directly; and -yet, ``() -> Int`` is a valid substitution of ``() -> T``, and a -caller of ``generateArray`` should be able to pass an arbitrary -``() -> Int`` in as the generator. - -Therefore, the representation of a formal type in a generic context -may differ from the representation of a substitution of that formal type. -We call such differences *abstraction differences*. - -SIL's type system is designed to make abstraction differences always -result in differences between SIL types. The goal is that a properly- -abstracted value should be correctly usable at any level of substitution. - -In order to achieve this, the formal type of a generic entity should -always be lowered using the abstraction pattern of its unsubstituted -formal type. For example, consider the following generic type: - -:: - - struct Generator { - var fn : () -> T - } - var intGen : Generator - -``intGen.fn`` has the substituted formal type ``() -> Int``, which -would normally lower to the type ``@callee_owned () -> Int``, i.e. -returning its result directly. But if that type is properly lowered -with the pattern of its unsubstituted type ``() -> T``, it becomes -``@callee_owned (@out Int) -> ()``. - -When a type is lowered using the abstraction pattern of an -unrestricted type, it is lowered as if the pattern were replaced with -a type sharing the same structure but replacing all materializable -types with fresh type variables. - -For example, if ``g`` has type ``Generator<(Int,Int) -> Float>``, ``g.fn`` is -lowered using the pattern ``() -> T``, which eventually causes ``(Int,Int) --> Float`` to be lowered using the pattern ``T``, which is the same as -lowering it with the pattern ``U -> V``; the result is that ``g.fn`` -has the following lowered type:: - - @callee_owned () -> @owned @callee_owned (@out Float, @in (Int,Int)) -> ()``. - -As another example, suppose that ``h`` has type -``Generator<(Int, @inout Int) -> Float>``. Neither ``(Int, @inout Int)`` -nor ``@inout Int`` are potential results of substitution because they -aren't materializable, so ``h.fn`` has the following lowered type:: - - @callee_owned () -> @owned @callee_owned (@out Float, @in Int, @inout Int) - -This system has the property that abstraction patterns are preserved -through repeated substitutions. That is, you can consider a lowered -type to encode an abstraction pattern; lowering ``T`` by ``R`` is -equivalent to lowering ``T`` by (``S`` lowered by ``R``). - -SILGen has procedures for converting values between abstraction -patterns. - -At present, only function and tuple types are changed by abstraction -differences. - -Legal SIL Types -``````````````` - -The type of a value in SIL shall be: - -- a loadable legal SIL type, ``$T``, - -- the address of a legal SIL type, ``$*T``, or - -- the address of local storage of a legal SIL type, ``$*@local_storage T``. - -A type ``T`` is a *legal SIL type* if: - -- it is a function type which satisfies the constraints (below) on - function types in SIL, - -- it is a tuple type whose element types are legal SIL types, - -- it is a legal Swift type that is not a function, tuple, or l-value type, or - -- it is a ``@box`` containing a legal SIL type. - -Note that types in other recursive positions in the type grammar are -still formal types. For example, the instance type of a metatype or -the type arguments of a generic type are still formal Swift types, not -lowered SIL types. - -Address Types -````````````` - -The *address of T* ``$*T`` is a pointer to memory containing a value -of any reference or value type ``$T``. This can be an internal -pointer into a data structure. Addresses of loadable types can be -loaded and stored to access values of those types. - -Addresses of address-only types (see below) can only be used with -instructions that manipulate their operands indirectly by address, such -as ``copy_addr`` or ``destroy_addr``, or as arguments to functions. -It is illegal to have a value of type ``$T`` if ``T`` is address-only. - -Addresses are not reference-counted pointers like class values are. They -cannot be retained or released. - -Address types are not *first-class*: they cannot appear in recursive -positions in type expressions. For example, the type ``$**T`` is not -a legal type. - -The address of an address cannot be directly taken. ``$**T`` is not a representable -type. Values of address type thus cannot be allocated, loaded, or stored -(though addresses can of course be loaded from and stored to). - -Addresses can be passed as arguments to functions if the corresponding -parameter is indirect. They cannot be returned. - -Local Storage Types -``````````````````` - -The *address of local storage for T* ``$*@local_storage T`` is a -handle to a stack allocation of a variable of type ``$T``. - -For many types, the handle for a stack allocation is simply the -allocated address itself. However, if a type is runtime-sized, the -compiler must emit code to potentially dynamically allocate memory. -SIL abstracts over such differences by using values of local-storage -type as the first result of ``alloc_stack`` and the operand of -``dealloc_stack``. - -Local-storage address types are not *first-class* in the same sense -that address types are not first-class. - -Box Types -````````` - -Captured local variables and the payloads of ``indirect`` value types are stored -on the heap. The type ``@box T`` is a reference-counted type that references -a box containing a mutable value of type ``T``. Boxes always use Swift-native -reference counting, so they can be queried for uniqueness and cast to the -``Builtin.NativeObject`` type. - -Function Types -`````````````` - -Function types in SIL are different from function types in Swift in a -number of ways: - -- A SIL function type may be generic. For example, accessing a - generic function with ``function_ref`` will give a value of - generic function type. - -- A SIL function type declares its conventional treatment of its - context value: - - - If it is ``@thin``, the function requires no context value. - - - If it is ``@callee_owned``, the context value is treated as an - owned direct parameter. - - - If it is ``@callee_guaranteed``, the context value is treated as - a guaranteed direct parameter. - - - Otherwise, the context value is treated as an unowned direct - parameter. - -- A SIL function type declares the conventions for its parameters, - including any implicit out-parameters. The parameters are written - as an unlabelled tuple; the elements of that tuple must be legal SIL - types, optionally decorated with one of the following convention - attributes. - - The value of an indirect parameter has type ``*T``; the value of a - direct parameter has type ``T``. - - - An ``@in`` parameter is indirect. The address must be of an - initialized object; the function is responsible for destroying - the value held there. - - - An ``@inout`` parameter is indirect. The address must be of an - initialized object, and the function must leave an initialized - object there upon exit. - - - An ``@out`` parameter is indirect. The address must be of an - uninitialized object; the function is responsible for initializing - a value there. If there is an ``@out`` parameter, it must be - the first parameter, and the direct result must be ``()``. - - - An ``@owned`` parameter is an owned direct parameter. - - - A ``@guaranteed`` parameter is a guaranteed direct parameter. - - - An ``@in_guaranteed`` parameter is indirect. The address must be of an - initialized object; both the caller and callee promise not to mutate the - pointee, allowing the callee to read it. - - - Otherwise, the parameter is an unowned direct parameter. - -- A SIL function type declares the convention for its direct result. - The result must be a legal SIL type. - - - An ``@owned`` result is an owned direct result. - - - An ``@autoreleased`` result is an autoreleased direct result. - - - Otherwise, the parameter is an unowned direct result. - -A direct parameter or result of trivial type must always be unowned. - -An owned direct parameter or result is transferred to the recipient, -which becomes responsible for destroying the value. This means that -the value is passed at +1. - -An unowned direct parameter or result is instantaneously valid at the -point of transfer. The recipient does not need to worry about race -conditions immediately destroying the value, but should copy it -(e.g. by ``strong_retain``\ ing an object pointer) if the value will be -needed sooner rather than later. - -A guaranteed direct parameter is like an unowned direct parameter -value, except that it is guaranteed by the caller to remain valid -throughout the execution of the call. This means that any -``strong_retain``, ``strong_release`` pairs in the callee on the -argument can be eliminated. - -An autoreleased direct result must have a type with a retainable -pointer representation. It may have been autoreleased, and the caller -should take action to reclaim that autorelease with -``strong_retain_autoreleased``. - -- The @noescape declaration attribute on Swift parameters (which is valid only - on parameters of function type, and is implied by the @autoclosure attribute) - is turned into a @noescape type attribute on SIL arguments. @noescape - indicates that the lifetime of the closure parameter will not be extended by - the callee (e.g. the pointer will not be stored in a global variable). It - corresponds to the LLVM "nocapture" attribute in terms of semantics (but is - limited to only work with parameters of function type in Swift). - -- SIL function types may provide an optional error result, written by - placing ``@error`` on a result. An error result is always - implicitly ``@owned``. Only functions with a native calling - convention may have an error result. - - A function with an error result cannot be called with ``apply``. - It must be called with ``try_apply``. - There is one exception to this rule: a function with an error result can be - called with ``apply [nothrow]`` if the compiler can prove that the function - does not actually throw. - - ``return`` produces a normal result of the function. To return - an error result, use ``throw``. - - Type lowering lowers the ``throws`` annotation on formal function - types into more concrete error propagation: - - - For native Swift functions, ``throws`` is turned into an error - result. - - - For non-native Swift functions, ``throws`` is turned in an - explicit error-handling mechanism based on the imported API. The - importer only imports non-native methods and types as ``throws`` - when it is possible to do this automatically. - -Properties of Types -``````````````````` - -SIL classifies types into additional subgroups based on ABI stability and -generic constraints: - -- *Loadable types* are types with a fully exposed concrete representation: - - * Reference types - * Builtin value types - * Fragile struct types in which all element types are loadable - * Tuple types in which all element types are loadable - * Class protocol types - * Archetypes constrained by a class protocol - - A *loadable aggregate type* is a tuple or struct type that is loadable. - - A *trivial type* is a loadable type with trivial value semantics. - Values of trivial type can be loaded and stored without any retain or - release operations and do not need to be destroyed. - -- *Runtime-sized types* are restricted value types for which the compiler - does not know the size of the type statically: - - * Resilient value types - * Fragile struct or tuple types that contain resilient types as elements at - any depth - * Archetypes not constrained by a class protocol - -- *Address-only types* are restricted value types which cannot be - loaded or otherwise worked with as SSA values: - - * Runtime-sized types - * Non-class protocol types - * @weak types - - Values of address-only type (“address-only values”) must reside in - memory and can only be referenced in SIL by address. Addresses of - address-only values cannot be loaded from or stored to. SIL provides - special instructions for indirectly manipulating address-only - values, such as ``copy_addr`` and ``destroy_addr``. - -Some additional meaningful categories of type: - -- A *heap object reference* type is a type whose representation consists of a - single strong-reference-counted pointer. This includes all class types, - the ``Builtin.ObjectPointer`` and ``Builtin.ObjCPointer`` types, and - archetypes that conform to one or more class protocols. -- A *reference type* is more general in that its low-level representation may - include additional global pointers alongside a strong-reference-counted - pointer. This includes all heap object reference types and adds - thick function types and protocol/protocol composition types that conform to - one or more class protocols. All reference types can be ``retain``-ed and - ``release``-d. Reference types also have *ownership semantics* for their - referenced heap object; see `Reference Counting`_ below. -- A type with *retainable pointer representation* is guaranteed to - be compatible (in the C sense) with the Objective-C ``id`` type. - The value at runtime may be ``nil``. This includes classes, - class metatypes, block functions, and class-bounded existentials with - only Objective-C-compatible protocol constraints, as well as one - level of ``Optional`` or ``ImplicitlyUnwrappedOptional`` applied to any of the - above. Types with retainable pointer representation can be returned - via the ``@autoreleased`` return convention. - -SILGen does not always map Swift function types one-to-one to SIL function -types. Function types are transformed in order to encode additional attributes: - -- The **convention** of the function, indicated by the - - .. parsed-literal:: - - @convention(*convention*) - - attribute. This is similar to the language-level ``@convention`` - attribute, though SIL extends the set of supported conventions with - additional distinctions not exposed at the language level: - - - ``@convention(thin)`` indicates a "thin" function reference, which uses - the Swift calling convention with no special "self" or "context" parameters. - - ``@convention(thick)`` indicates a "thick" function reference, which - uses the Swift calling convention and carries a reference-counted context - object used to represent captures or other state required by the function. - - ``@convention(block)`` indicates an Objective-C compatible block reference. - The function value is represented as a reference to the block object, - which is an ``id``-compatible Objective-C object that embeds its invocation - function within the object. The invocation function uses the C calling - convention. - - ``@convention(c)`` indicates a C function reference. The function value - carries no context and uses the C calling convention. - - ``@convention(objc_method)`` indicates an Objective-C method implementation. - The function uses the C calling convention, with the SIL-level ``self`` - parameter (by SIL convention mapped to the final formal parameter) - mapped to the ``self`` and ``_cmd`` arguments of the implementation. - - ``@convention(method)`` indicates a Swift instance method implementation. - The function uses the Swift calling convention, using the special ``self`` - parameter. - - ``@convention(witness_method)`` indicates a Swift protocol method - implementation. The function's polymorphic convention is emitted in such - a way as to guarantee that it is polymorphic across all possible - implementors of the protocol. - -- The **fully uncurried representation** of the function type, with - all of the curried argument clauses flattened into a single argument - clause. For instance, a curried function ``func foo(x:A)(y:B) -> C`` - might be emitted as a function of type ``((y:B), (x:A)) -> C``. The - exact representation depends on the function's `calling - convention`_, which determines the exact ordering of currying - clauses. Methods are treated as a form of curried function. - -Layout Compatible Types -``````````````````````` - -(This section applies only to Swift 1.0 and will hopefully be obviated in -future releases.) - -SIL tries to be ignorant of the details of type layout, and low-level -bit-banging operations such as pointer casts are generally undefined. However, -as a concession to implementation convenience, some types are allowed to be -considered **layout compatible**. Type ``T`` is *layout compatible* with type -``U`` iff: - -- an address of type ``$*U`` can be cast by - ``address_to_pointer``/``pointer_to_address`` to ``$*T`` and a valid value - of type ``T`` can be loaded out (or indirectly used, if ``T`` is address- - only), -- if ``T`` is a nontrivial type, then ``retain_value``/``release_value`` of - the loaded ``T`` value is equivalent to ``retain_value``/``release_value`` of - the original ``U`` value. - -This is not always a commutative relationship; ``T`` can be layout-compatible -with ``U`` whereas ``U`` is not layout-compatible with ``T``. If the layout -compatible relationship does extend both ways, ``T`` and ``U`` are -**commutatively layout compatible**. It is however always transitive; if ``T`` -is layout-compatible with ``U`` and ``U`` is layout-compatible with ``V``, then -``T`` is layout-compatible with ``V``. All types are layout-compatible with -themselves. - -The following types are considered layout-compatible: - -- ``Builtin.RawPointer`` is commutatively layout compatible with all heap - object reference types, and ``Optional`` of heap object reference types. - (Note that ``RawPointer`` is a trivial type, so does not have ownership - semantics.) -- ``Builtin.RawPointer`` is commutatively layout compatible with - ``Builtin.Word``. -- Structs containing a single stored property are commutatively layout - compatible with the type of that property. -- A heap object reference is commutatively layout compatible with any type - that can correctly reference the heap object. For instance, given a class - ``B`` and a derived class ``D`` inheriting from ``B``, a value of - type ``B`` referencing an instance of type ``D`` is layout compatible with - both ``B`` and ``D``, as well as ``Builtin.NativeObject`` and - ``Builtin.UnknownObject``. It is not layout compatible with an unrelated class - type ``E``. -- For payloaded enums, the payload type of the first payloaded case is - layout-compatible with the enum (*not* commutatively). - -Values and Operands -~~~~~~~~~~~~~~~~~~~ -:: - - sil-identifier ::= [A-Za-z_0-9]+ - sil-value-name ::= '%' sil-identifier - sil-value ::= sil-value-name ('#' [0-9]+)? - sil-value ::= 'undef' - sil-operand ::= sil-value ':' sil-type - -SIL values are introduced with the ``%`` sigil and named by an -alphanumeric identifier, which references the instruction or basic block -argument that produces the value. SIL values may also refer to the keyword -'undef', which is a value of undefined contents. -In SIL, a single instruction may produce multiple values. Operands that refer -to multiple-value instructions choose the value by following the ``%name`` with -``#`` and the index of the value. For example:: - - // alloc_box produces two values--the refcounted pointer %box#0, and the - // value address %box#1 - %box = alloc_box $Int64 - // Refer to the refcounted pointer - strong_retain %box#0 : $@box Int64 - // Refer to the address - store %value to %box#1 : $*Int64 - -Unlike LLVM IR, SIL instructions that take value operands *only* accept -value operands. References to literal constants, functions, global variables, or -other entities require specialized instructions such as ``integer_literal``, -``function_ref``, ``global_addr``, etc. - -Functions -~~~~~~~~~ -:: - - decl ::= sil-function - sil-function ::= 'sil' sil-linkage? sil-function-name ':' sil-type - '{' sil-basic-block+ '}' - sil-function-name ::= '@' [A-Za-z_0-9]+ - -SIL functions are defined with the ``sil`` keyword. SIL function names -are introduced with the ``@`` sigil and named by an alphanumeric -identifier. This name will become the LLVM IR name for the function, -and is usually the mangled name of the originating Swift declaration. -The ``sil`` syntax declares the function's name and SIL type, and -defines the body of the function inside braces. The declared type must -be a function type, which may be generic. - -Basic Blocks -~~~~~~~~~~~~ -:: - - sil-basic-block ::= sil-label sil-instruction-def* sil-terminator - sil-label ::= sil-identifier ('(' sil-argument (',' sil-argument)* ')')? ':' - sil-argument ::= sil-value-name ':' sil-type - - sil-instruction-def ::= (sil-value-name '=')? sil-instruction - -A function body consists of one or more basic blocks that correspond -to the nodes of the function's control flow graph. Each basic block -contains one or more instructions and ends with a terminator -instruction. The function's entry point is always the first basic -block in its body. - -In SIL, basic blocks take arguments, which are used as an alternative to LLVM's -phi nodes. Basic block arguments are bound by the branch from the predecessor -block:: - - sil @iif : $(Builtin.Int1, Builtin.Int64, Builtin.Int64) -> Builtin.Int64 { - bb0(%cond : $Builtin.Int1, %ifTrue : $Builtin.Int64, %ifFalse : $Builtin.Int64): - cond_br %cond : $Builtin.Int1, then, else - then: - br finish(%ifTrue : $Builtin.Int64) - else: - br finish(%ifFalse : $Builtin.Int64) - finish(%result : $Builtin.Int64): - return %result : $Builtin.Int64 - } - -Arguments to the entry point basic block, which has no predecessor, -are bound by the function's caller:: - - sil @foo : $(Int) -> Int { - bb0(%x : $Int): - %1 = return %x : $Int - } - - sil @bar : $(Int, Int) -> () { - bb0(%x : $Int, %y : $Int): - %foo = function_ref @foo - %1 = apply %foo(%x) : $(Int) -> Int - %2 = apply %foo(%y) : $(Int) -> Int - %3 = tuple () - %4 = return %3 : $() - } - -Declaration References -~~~~~~~~~~~~~~~~~~~~~~ -:: - - sil-decl-ref ::= '#' sil-identifier ('.' sil-identifier)* sil-decl-subref? - sil-decl-subref ::= '!' sil-decl-subref-part ('.' sil-decl-uncurry-level)? ('.' sil-decl-lang)? - sil-decl-subref ::= '!' sil-decl-uncurry-level ('.' sil-decl-lang)? - sil-decl-subref ::= '!' sil-decl-lang - sil-decl-subref-part ::= 'getter' - sil-decl-subref-part ::= 'setter' - sil-decl-subref-part ::= 'allocator' - sil-decl-subref-part ::= 'initializer' - sil-decl-subref-part ::= 'enumelt' - sil-decl-subref-part ::= 'destroyer' - sil-decl-subref-part ::= 'deallocator' - sil-decl-subref-part ::= 'globalaccessor' - sil-decl-subref-part ::= 'ivardestroyer' - sil-decl-subref-part ::= 'ivarinitializer' - sil-decl-subref-part ::= 'defaultarg' '.' [0-9]+ - sil-decl-uncurry-level ::= [0-9]+ - sil-decl-lang ::= 'foreign' - -Some SIL instructions need to reference Swift declarations directly. These -references are introduced with the ``#`` sigil followed by the fully qualified -name of the Swift declaration. Some Swift declarations are -decomposed into multiple entities at the SIL level. These are distinguished by -following the qualified name with ``!`` and one or more ``.``-separated component -entity discriminators: - -- ``getter``: the getter function for a ``var`` declaration -- ``setter``: the setter function for a ``var`` declaration -- ``allocator``: a ``struct`` or ``enum`` constructor, or a ``class``\ 's *allocating constructor* -- ``initializer``: a ``class``\ 's *initializing constructor* -- ``enumelt``: a member of a ``enum`` type. -- ``destroyer``: a class's destroying destructor -- ``deallocator``: a class's deallocating destructor -- ``globalaccessor``: the addressor function for a global variable -- ``ivardestroyer``: a class's ivar destroyer -- ``ivarinitializer``: a class's ivar initializer -- ``defaultarg.``\ *n*: the default argument-generating function for - the *n*\ -th argument of a Swift ``func`` -- ``foreign``: a specific entry point for C/objective-C interoperability - -Methods and curried function definitions in Swift also have multiple -"uncurry levels" in SIL, representing the function at each possible -partial application level. For a curried function declaration:: - - // Module example - func foo(x:A)(y:B)(z:C) -> D - -The declaration references and types for the different uncurry levels are as -follows:: - - #example.foo!0 : $@thin (x:A) -> (y:B) -> (z:C) -> D - #example.foo!1 : $@thin ((y:B), (x:A)) -> (z:C) -> D - #example.foo!2 : $@thin ((z:C), (y:B), (x:A)) -> D - -The deepest uncurry level is referred to as the **natural uncurry level**. In -this specific example, the reference at the natural uncurry level is -``#example.foo!2``. Note that the uncurried argument clauses are composed -right-to-left, as specified in the `calling convention`_. For uncurry levels -less than the uncurry level, the entry point itself is ``@thin`` but returns a -thick function value carrying the partially applied arguments for its context. - -`Dynamic dispatch`_ instructions such as ``class method`` require their method -declaration reference to be uncurried to at least uncurry level 1 (which applies -both the "self" argument and the method arguments), because uncurry level zero -represents the application of the method to its "self" argument, as in -``foo.method``, which is where the dynamic dispatch semantically occurs -in Swift. - -Linkage -~~~~~~~ -:: - - sil-linkage ::= 'public' - sil-linkage ::= 'hidden' - sil-linkage ::= 'shared' - sil-linkage ::= 'private' - sil-linkage ::= 'public_external' - sil-linkage ::= 'hidden_external' - -A linkage specifier controls the situations in which two objects in -different SIL modules are *linked*, i.e. treated as the same object. - -A linkage is *external* if it ends with the suffix ``external``. An -object must be a definition if its linkage is not external. - -All functions, global variables, and witness tables have linkage. -The default linkage of a definition is ``public``. The default linkage of a -declaration is ``public_external``. (These may eventually change to ``hidden`` -and ``hidden_external``, respectively.) - -On a global variable, an external linkage is what indicates that the -variable is not a definition. A variable lacking an explicit linkage -specifier is presumed a definition (and thus gets the default linkage -for definitions, ``public``.) - -Definition of the *linked* relation -``````````````````````````````````` - -Two objects are linked if they have the same name and are mutually -visible: - - - An object with ``public`` or ``public_external`` linkage is always - visible. - - - An object with ``hidden``, ``hidden_external``, or ``shared`` - linkage is visible only to objects in the same Swift module. - - - An object with ``private`` linkage is visible only to objects in - the same SIL module. - -Note that the *linked* relationship is an equivalence relation: it is -reflexive, symmetric, and transitive. - -Requirements on linked objects -`````````````````````````````` - -If two objects are linked, they must have the same type. - -If two objects are linked, they must have the same linkage, except: - - - A ``public`` object may be linked to a ``public_external`` object. - - - A ``hidden`` object may be linked to a ``hidden_external`` object. - -If two objects are linked, at most one may be a definition, unless: - - - both objects have ``shared`` linkage or - - - at least one of the objects has an external linkage. - -If two objects are linked, and both are definitions, then the -definitions must be semantically equivalent. This equivalence may -exist only on the level of user-visible semantics of well-defined -code; it should not be taken to guarantee that the linked definitions -are exactly operationally equivalent. For example, one definition of -a function might copy a value out of an address parameter, while -another may have had an analysis applied to prove that said value is -not needed. - -If an object has any uses, then it must be linked to a definition -with non-external linkage. - -Summary -``````` - - - ``public`` definitions are unique and visible everywhere in the - program. In LLVM IR, they will be emitted with ``external`` - linkage and ``default`` visibility. - - - ``hidden`` definitions are unique and visible only within the - current Swift module. In LLVM IR, they will be emitted with - ``external`` linkage and ``hidden`` visibility. - - - ``private`` definitions are unique and visible only within the - current SIL module. In LLVM IR, they will be emitted with - ``private`` linkage. - - - ``shared`` definitions are visible only within the current Swift - module. They can be linked only with other ``shared`` - definitions, which must be equivalent; therefore, they only need - to be emitted if actually used. In LLVM IR, they will be emitted - with ``linkonce_odr`` linkage and ``hidden`` visibility. - - - ``public_external`` and ``hidden_external`` objects always have - visible definitions somewhere else. If this object nonetheless - has a definition, it's only for the benefit of optimization or - analysis. In LLVM IR, declarations will have ``external`` linkage - and definitions (if actually emitted as definitions) will have - ``available_externally`` linkage. - - -VTables -~~~~~~~ -:: - - decl ::= sil-vtable - sil-vtable ::= 'sil_vtable' identifier '{' sil-vtable-entry* '}' - - sil-vtable-entry ::= sil-decl-ref ':' sil-function-name - -SIL represents dynamic dispatch for class methods using the `class_method`_, -`super_method`_, and `dynamic_method`_ instructions. The potential destinations -for these dispatch operations are tracked in ``sil_vtable`` declarations for -every class type. The declaration contains a mapping from every method of the -class (including those inherited from its base class) to the SIL function that -implements the method for that class:: - - class A { - func foo() - func bar() - func bas() - } - - sil @A_foo : $@thin (@owned A) -> () - sil @A_bar : $@thin (@owned A) -> () - sil @A_bas : $@thin (@owned A) -> () - - sil_vtable A { - #A.foo!1: @A_foo - #A.bar!1: @A_bar - #A.bas!1: @A_bas - } - - class B : A { - func bar() - } - - sil @B_bar : $@thin (@owned B) -> () - - sil_vtable B { - #A.foo!1: @A_foo - #A.bar!1: @B_bar - #A.bas!1: @A_bas - } - - class C : B { - func bas() - } - - sil @C_bas : $@thin (@owned C) -> () - - sil_vtable C { - #A.foo!1: @A_foo - #A.bar!1: @B_bar - #A.bas!1: @C_bas - } - -Note that the declaration reference in the vtable is to the least-derived method -visible through that class (in the example above, ``B``'s vtable references -``A.bar`` and not ``B.bar``, and ``C``'s vtable references ``A.bas`` and not -``C.bas``). The Swift AST maintains override relationships between declarations -that can be used to look up overridden methods in the SIL vtable for a derived -class (such as ``C.bas`` in ``C``'s vtable). - -Witness Tables -~~~~~~~~~~~~~~ -:: - - decl ::= sil-witness-table - sil-witness-table ::= 'sil_witness_table' sil-linkage? - normal-protocol-conformance '{' sil-witness-entry* '}' - -SIL encodes the information needed for dynamic dispatch of generic types into -witness tables. This information is used to produce runtime dispatch tables when -generating binary code. It can also be used by SIL optimizations to specialize -generic functions. A witness table is emitted for every declared explicit -conformance. Generic types share one generic witness table for all of their -instances. Derived classes inherit the witness tables of their base class. - -:: - - protocol-conformance ::= normal-protocol-conformance - protocol-conformance ::= 'inherit' '(' protocol-conformance ')' - protocol-conformance ::= 'specialize' '<' substitution* '>' - '(' protocol-conformance ')' - protocol-conformance ::= 'dependent' - normal-protocol-conformance ::= identifier ':' identifier 'module' identifier - -Witness tables are keyed by *protocol conformance*, which is a unique identifier -for a concrete type's conformance to a protocol. - -- A *normal protocol conformance* - names a (potentially unbound generic) type, the protocol it conforms to, and - the module in which the type or extension declaration that provides the - conformance appears. These correspond 1:1 to protocol conformance declarations - in the source code. -- If a derived class conforms to a protocol through inheritance from its base - class, this is represented by an *inherited protocol conformance*, which - simply references the protocol conformance for the base class. -- If an instance of a generic type conforms to a protocol, it does so with a - *specialized conformance*, which provides the generic parameter bindings - to the normal conformance, which should be for a generic type. - -Witness tables are only directly associated with normal conformances. -Inherited and specialized conformances indirectly reference the witness table of -the underlying normal conformance. - -:: - - sil-witness-entry ::= 'base_protocol' identifier ':' protocol-conformance - sil-witness-entry ::= 'method' sil-decl-ref ':' sil-function-name - sil-witness-entry ::= 'associated_type' identifier - sil-witness-entry ::= 'associated_type_protocol' - '(' identifier ':' identifier ')' ':' protocol-conformance - -Witness tables consist of the following entries: - -- *Base protocol entries* provide references to the protocol conformances that - satisfy the witnessed protocols' inherited protocols. -- *Method entries* map a method requirement of the protocol to a SIL function - that implements that method for the witness type. One method entry must exist - for every required method of the witnessed protocol. -- *Associated type entries* map an associated type requirement of the protocol - to the type that satisfies that requirement for the witness type. Note that - the witness type is a source-level Swift type and not a SIL type. One - associated type entry must exist for every required associated type of the - witnessed protocol. -- *Associated type protocol entries* map a protocol requirement on an associated - type to the protocol conformance that satisfies that requirement for the - associated type. - -Global Variables -~~~~~~~~~~~~~~~~ -:: - - decl ::= sil-global-variable - sil-global-variable ::= 'sil_global' sil-linkage identifier ':' sil-type - -SIL representation of a global variable. - -FIXME: to be written. - -Dataflow Errors ---------------- - -*Dataflow errors* may exist in raw SIL. Swift's semantics defines these -conditions as errors, so they must be diagnosed by diagnostic -passes and must not exist in canonical SIL. - -Definitive Initialization -~~~~~~~~~~~~~~~~~~~~~~~~~ - -Swift requires that all local variables be initialized before use. In -constructors, all instance variables of a struct, enum, or class type must -be initialized before the object is used and before the constructor is returned -from. - -Unreachable Control Flow -~~~~~~~~~~~~~~~~~~~~~~~~ - -The ``unreachable`` terminator is emitted in raw SIL to mark incorrect control -flow, such as a non-``Void`` function failing to ``return`` a value, or a -``switch`` statement failing to cover all possible values of its subject. -The guaranteed dead code elimination pass can eliminate truly unreachable -basic blocks, or ``unreachable`` instructions may be dominated by applications -of ``@noreturn`` functions. An ``unreachable`` instruction that survives -guaranteed DCE and is not immediately preceded by a ``@noreturn`` -application is a dataflow error. - -Runtime Failure ---------------- - -Some operations, such as failed unconditional `checked conversions`_ or the -``Builtin.trap`` compiler builtin, cause a *runtime failure*, which -unconditionally terminates the current actor. If it can be proven that a -runtime failure will occur or did occur, runtime failures may be reordered so -long as they remain well-ordered relative to operations external to the actor -or the program as a whole. For instance, with overflow checking on integer -arithmetic enabled, a simple ``for`` loop that reads inputs in from one or more -arrays and writes outputs to another array, all local -to the current actor, may cause runtime failure in the update operations:: - - // Given unknown start and end values, this loop may overflow - for var i = unknownStartValue; i != unknownEndValue; ++i { - ... - } - -It is permitted to hoist the overflow check and associated runtime failure out -of the loop itself and check the bounds of the loop prior to entering it, so -long as the loop body has no observable effect outside of the current actor. - -Undefined Behavior ------------------- - -Incorrect use of some operations is *undefined behavior*, such as invalid -unchecked casts involving ``Builtin.RawPointer`` types, or use of compiler -builtins that lower to LLVM instructions with undefined behavior at the LLVM -level. A SIL program with undefined behavior is meaningless, much like undefined -behavior in C, and has no predictable semantics. Undefined behavior should not -be triggered by valid SIL emitted by a correct Swift program using a correct -standard library, but cannot in all cases be diagnosed or verified at the SIL -level. - -Calling Convention ------------------- - -This section describes how Swift functions are emitted in SIL. - -Swift Calling Convention @cc(swift) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The Swift calling convention is the one used by default for native Swift -functions. - -Tuples in the input type of the function are recursively destructured into -separate arguments, both in the entry point basic block of the callee, and -in the ``apply`` instructions used by callers:: - - func foo(x:Int, y:Int) - - sil @foo : $(x:Int, y:Int) -> () { - entry(%x : $Int, %y : $Int): - ... - } - - func bar(x:Int, y:(Int, Int)) - - sil @bar : $(x:Int, y:(Int, Int)) -> () { - entry(%x : $Int, %y0 : $Int, %y1 : $Int): - ... - } - - func call_foo_and_bar() { - foo(1, 2) - bar(4, (5, 6)) - } - - sil @call_foo_and_bar : $() -> () { - entry: - ... - %foo = function_ref @foo : $(x:Int, y:Int) -> () - %foo_result = apply %foo(%1, %2) : $(x:Int, y:Int) -> () - ... - %bar = function_ref @bar : $(x:Int, y:(Int, Int)) -> () - %bar_result = apply %bar(%4, %5, %6) : $(x:Int, y:(Int, Int)) -> () - } - -Calling a function with trivial value types as inputs and outputs -simply passes the arguments by value. This Swift function:: - - func foo(x:Int, y:Float) -> UnicodeScalar - - foo(x, y) - -gets called in SIL as:: - - %foo = constant_ref $(Int, Float) -> UnicodeScalar, @foo - %z = apply %foo(%x, %y) : $(Int, Float) -> UnicodeScalar - -Reference Counts -```````````````` - -*NOTE* This section only is speaking in terms of rules of thumb. The -actual behavior of arguments with respect to arguments is defined by -the argument's convention attribute (e.g. ``@owned``), not the -calling convention itself. - -Reference type arguments are passed in at +1 retain count and consumed -by the callee. A reference type return value is returned at +1 and -consumed by the caller. Value types with reference type components -have their reference type components each retained and released the -same way. This Swift function:: - - class A {} - - func bar(x:A) -> (Int, A) { ... } - - bar(x) - -gets called in SIL as:: - - %bar = function_ref @bar : $(A) -> (Int, A) - strong_retain %x : $A - %z = apply %bar(%x) : $(A) -> (Int, A) - // ... use %z ... - %z_1 = tuple_extract %z : $(Int, A), 1 - strong_release %z_1 - -When applying a thick function value as a callee, the function value is also -consumed at +1 retain count. - -Address-Only Types -`````````````````` - -For address-only arguments, the caller allocates a copy and passes the address -of the copy to the callee. The callee takes ownership of the copy and is -responsible for destroying or consuming the value, though the caller must still -deallocate the memory. For address-only return values, the -caller allocates an uninitialized buffer and passes its address as the first -argument to the callee. The callee must initialize this buffer before -returning. This Swift function:: - - @API struct A {} - - func bas(x:A, y:Int) -> A { return x } - - var z = bas(x, y) - // ... use z ... - -gets called in SIL as:: - - %bas = function_ref @bas : $(A, Int) -> A - %z = alloc_stack $A - %x_arg = alloc_stack $A - copy_addr %x to [initialize] %x_arg : $*A - apply %bas(%z, %x_arg, %y) : $(A, Int) -> A - dealloc_stack %x_arg : $*A // callee consumes %x.arg, caller deallocs - // ... use %z ... - destroy_addr %z : $*A - dealloc_stack stack %z : $*A - -The implementation of ``@bas`` is then responsible for consuming ``%x_arg`` and -initializing ``%z``. - -Tuple arguments are destructured regardless of the -address-only-ness of the tuple type. The destructured fields are passed -individually according to the above convention. This Swift function:: - - @API struct A {} - - func zim(x:Int, y:A, (z:Int, w:(A, Int))) - - zim(x, y, (z, w)) - -gets called in SIL as:: - - %zim = function_ref @zim : $(x:Int, y:A, (z:Int, w:(A, Int))) -> () - %y_arg = alloc_stack $A - copy_addr %y to [initialize] %y_arg : $*A - %w_0_addr = element_addr %w : $*(A, Int), 0 - %w_0_arg = alloc_stack $A - copy_addr %w_0_addr to [initialize] %w_0_arg : $*A - %w_1_addr = element_addr %w : $*(A, Int), 1 - %w_1 = load %w_1_addr : $*Int - apply %zim(%x, %y_arg, %z, %w_0_arg, %w_1) : $(x:Int, y:A, (z:Int, w:(A, Int))) -> () - dealloc_stack %w_0_arg - dealloc_stack %y_arg - -Variadic Arguments -`````````````````` - -Variadic arguments and tuple elements are packaged into an array and passed as -a single array argument. This Swift function:: - - func zang(x:Int, (y:Int, z:Int...), v:Int, w:Int...) - - zang(x, (y, z0, z1), v, w0, w1, w2) - -gets called in SIL as:: - - %zang = function_ref @zang : $(x:Int, (y:Int, z:Int...), v:Int, w:Int...) -> () - %zs = <> - %ws = <> - apply %zang(%x, %y, %zs, %v, %ws) : $(x:Int, (y:Int, z:Int...), v:Int, w:Int...) -> () - -Function Currying -````````````````` - -Curried function definitions in Swift emit multiple SIL entry points, one for -each "uncurry level" of the function. When a function is uncurried, its -outermost argument clauses are combined into a tuple in right-to-left order. -For the following declaration:: - - func curried(x:A)(y:B)(z:C)(w:D) -> Int {} - -The types of the SIL entry points are as follows:: - - sil @curried_0 : $(x:A) -> (y:B) -> (z:C) -> (w:D) -> Int { ... } - sil @curried_1 : $((y:B), (x:A)) -> (z:C) -> (w:D) -> Int { ... } - sil @curried_2 : $((z:C), (y:B), (x:A)) -> (w:D) -> Int { ... } - sil @curried_3 : $((w:D), (z:C), (y:B), (x:A)) -> Int { ... } - -@inout Arguments -```````````````` - -``@inout`` arguments are passed into the entry point by address. The callee -does not take ownership of the referenced memory. The referenced memory must -be initialized upon function entry and exit. If the ``@inout`` argument -refers to a fragile physical variable, then the argument is the address of that -variable. If the ``@inout`` argument refers to a logical property, then the -argument is the address of a caller-owned writeback buffer. It is the caller's -responsibility to initialize the buffer by storing the result of the property -getter prior to calling the function and to write back to the property -on return by loading from the buffer and invoking the setter with the final -value. This Swift function:: - - func inout(x:@inout Int) { - x = 1 - } - -gets lowered to SIL as:: - - sil @inout : $(@inout Int) -> () { - entry(%x : $*Int): - %1 = integer_literal 1 : $Int - store %1 to %x - return - } - -Swift Method Calling Convention @cc(method) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The method calling convention is currently identical to the freestanding -function convention. Methods are considered to be curried functions, taking -the "self" argument as their outer argument clause, and the method arguments -as the inner argument clause(s). When uncurried, the "self" argument is thus -passed last:: - - struct Foo { - func method(x:Int) -> Int {} - } - - sil @Foo_method_1 : $((x : Int), @inout Foo) -> Int { ... } - -Witness Method Calling Convention @cc(witness_method) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The witness method calling convention is used by protocol witness methods in -`witness tables`_. It is identical to the ``method`` calling convention -except that its handling of generic type parameters. For non-witness methods, -the machine-level convention for passing type parameter metadata may be -arbitrarily dependent on static aspects of the function signature, but because -witnesses must be polymorphically dispatchable on their ``Self`` type, -the ``Self``-related metadata for a witness must be passed in a maximally -abstracted manner. - -C Calling Convention @cc(cdecl) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -In Swift's C module importer, C types are always mapped to Swift types -considered trivial by SIL. SIL does not concern itself with platform -ABI requirements for indirect return, register vs. stack passing, etc.; C -function arguments and returns in SIL are always by value regardless of the -platform calling convention. - -SIL (and therefore Swift) cannot currently invoke variadic C functions. - -Objective-C Calling Convention @cc(objc_method) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Reference Counts -```````````````` - -Objective-C methods use the same argument and return value ownership rules as -ARC Objective-C. Selector families and the ``ns_consumed``, -``ns_returns_retained``, etc. attributes from imported Objective-C definitions -are honored. - -Applying a ``@convention(block)`` value does not consume the block. - -Method Currying -``````````````` - -In SIL, the "self" argument of an Objective-C method is uncurried to the last -argument of the uncurried type, just like a native Swift method:: - - @objc class NSString { - func stringByPaddingToLength(Int) withString(NSString) startingAtIndex(Int) - } - - sil @NSString_stringByPaddingToLength_withString_startingAtIndex \ - : $((Int, NSString, Int), NSString) - -That ``self`` is passed as the first argument at the IR level is abstracted -away in SIL, as is the existence of the ``_cmd`` selector argument. - -Type Based Alias Analysis -------------------------- - -SIL supports two types of Type Based Alias Analysis (TBAA): Class TBAA and -Typed Access TBAA. - -Class TBAA -~~~~~~~~~~ - -Class instances and other *heap object references* are pointers at the -implementation level, but unlike SIL addresses, they are first class values and -can be ``capture``-d and alias. Swift, however, is memory-safe and statically -typed, so aliasing of classes is constrained by the type system as follows: - -* A ``Builtin.NativeObject`` may alias any native Swift heap object, - including a Swift class instance, a box allocated by ``alloc_box``, - or a thick function's closure context. - It may not alias natively Objective-C class instances. -* A ``Builtin.UnknownObject`` may alias any class instance, whether Swift or - Objective-C, but may not alias non-class-instance heap objects. -* Two values of the same class type ``$C`` may alias. Two values of related - class type ``$B`` and ``$D``, where there is a subclass relationship between - ``$B`` and ``$D``, may alias. Two values of unrelated class types may not - alias. This includes different instantiations of a generic class type, such - as ``$C`` and ``$C``, which currently may never alias. -* Without whole-program visibility, values of archetype or protocol type must - be assumed to potentially alias any class instance. Even if it is locally - apparent that a class does not conform to that protocol, another component - may introduce a conformance by an extension. Similarly, a generic class - instance, such as ``$C`` for archetype ``T``, must be assumed to - potentially alias concrete instances of the generic type, such as - ``$C``, because ``Int`` is a potential substitution for ``T``. - -Typed Access TBAA -~~~~~~~~~~~~~~~~~ - -Define a *typed access* of an address or reference as one of the following: - -* Any instruction that performs a typed read or write operation upon the memory - at the given location (e.x. ``load``, ``store``). -* Any instruction that yields a typed offset of the pointer by performing a - typed projection operation (e.x. ``ref_element_addr``, - ``tuple_element_addr``). - -It is undefined behavior to perform a typed access to an address or reference if -the stored object or referent is not an allocated object of the relevant type. - -This allows the optimizer to assume that two addresses cannot alias if there -does not exist a substitution of archetypes that could cause one of the types to -be the type of a subobject of the other. Additionally, this applies to the types -of the values from which the addresses were derived, ignoring "blessed" -alias-introducing operations such as ``pointer_to_address``, the ``bitcast`` -intrinsic, and the ``inttoptr`` intrinsic. - -Value Dependence ----------------- - -In general, analyses can assume that independent values are -independently assured of validity. For example, a class method may -return a class reference:: - - bb0(%0 : $MyClass): - %1 = class_method %0 : $MyClass, #MyClass.foo!1 - %2 = apply %1(%0) : $@cc(method) @thin (@guaranteed MyClass) -> @owned MyOtherClass - // use of %2 goes here; no use of %1 - strong_release %2 : $MyOtherClass - strong_release %1 : $MyClass - -The optimizer is free to move the release of ``%1`` to immediately -after the call here, because ``%2`` can be assumed to be an -independently-managed value, and because Swift generally permits the -reordering of destructors. - -However, some instructions do create values that are intrinsically -dependent on their operands. For example, the result of -``ref_element_addr`` will become a dangling pointer if the base is -released too soon. This is captured by the concept of *value dependence*, -and any transformation which can reorder of destruction of a value -around another operation must remain conscious of it. - -A value ``%1`` is said to be *value-dependent* on a value ``%0`` if: - -- ``%1`` is the result and ``%0`` is the first operand of one of the - following instructions: - - - ``ref_element_addr`` - - ``struct_element_addr`` - - ``tuple_element_addr`` - - ``unchecked_take_enum_data_addr`` - - ``pointer_to_address`` - - ``address_to_pointer`` - - ``index_addr`` - - ``index_raw_pointer`` - - possibly some other conversions - -- ``%1`` is the result of ``mark_dependence`` and ``%0`` is either of - the operands. - -- ``%1`` is the value address of an allocation instruction of which - ``%0`` is the local storage token or box reference. - -- ``%1`` is the result of a ``struct``, ``tuple``, or ``enum`` - instruction and ``%0`` is an operand. - -- ``%1`` is the result of projecting out a subobject of ``%0`` - with ``tuple_extract``, ``struct_extract``, ``unchecked_enum_data``, - ``select_enum``, or ``select_enum_addr``. - -- ``%1`` is the result of ``select_value`` and ``%0`` is one of the cases. - -- ``%1`` is a basic block parameter and ``%0`` is the corresponding - argument from a branch to that block. - -- ``%1`` is the result of a ``load`` from ``%0``. However, the value - dependence is cut after the first attempt to manage the value of - ``%1``, e.g. by retaining it. - -- Transitivity: there exists a value ``%2`` which ``%1`` depends on - and which depends on ``%0``. However, transitivity does not apply - to different subobjects of a struct, tuple, or enum. - -Note, however, that an analysis is not required to track dependence -through memory. Nor is it required to consider the possibility of -dependence being established "behind the scenes" by opaque code, such -as by a method returning an unsafe pointer to a class property. The -dependence is required to be locally obvious in a function's SIL -instructions. Precautions must be taken against this either by SIL -generators (by using ``mark_dependence`` appropriately) or by the user -(by using the appropriate intrinsics and attributes with unsafe -language or library features). - -Only certain types of SIL value can carry value-dependence: - -- SIL address types -- unmanaged pointer types: - - - ``@sil_unmanaged`` types - - ``Builtin.RawPointer`` - - aggregates containing such a type, such as ``UnsafePointer``, - possibly recursively - -- non-trivial types (but they can be independently managed) - -This rule means that casting a pointer to an integer type breaks -value-dependence. This restriction is necessary so that reading an -``Int`` from a class doesn't force the class to be kept around! -A class holding an unsafe reference to an object must use some -sort of unmanaged pointer type to do so. - -This rule does not include generic or resilient value types which -might contain unmanaged pointer types. Analyses are free to assume -that e.g. a ``copy_addr`` of a generic or resilient value type yields -an independently-managed value. The extension of value dependence to -types containing obvious unmanaged pointer types is an affordance to -make the use of such types more convenient; it does not shift the -ultimate responsibility for assuring the safety of unsafe -language/library features away from the user. - -Instruction Set ---------------- - -Allocation and Deallocation -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -These instructions allocate and deallocate memory. - -alloc_stack -``````````` -:: - - sil-instruction ::= 'alloc_stack' sil-type - - %1 = alloc_stack $T - // %1#0 has type $*@local_storage T - // %1#1 has type $*T - -Allocates uninitialized memory that is sufficiently aligned on the stack -to contain a value of type ``T``. The first result of the instruction -is a local-storage handle suitable for passing to ``dealloc_stack``. -The second result of the instruction is the address of the allocated memory. - -``alloc_stack`` marks the start of the lifetime of the value; the -allocation must be balanced with a ``dealloc_stack`` instruction to -mark the end of its lifetime. All ``alloc_stack`` allocations must be -deallocated prior to returning from a function. If a block has multiple -predecessors, the stack height and order of allocations must be consistent -coming from all predecessor blocks. ``alloc_stack`` allocations must be -deallocated in last-in, first-out stack order. - -The memory is not retainable. To allocate a retainable box for a value -type, use ``alloc_box``. - -alloc_ref -````````` -:: - - sil-instruction ::= 'alloc_ref' ('[' 'objc' ']')? ('[' 'stack' ']')? sil-type - - %1 = alloc_ref [stack] $T - // $T must be a reference type - // %1 has type $T - -Allocates an object of reference type ``T``. The object will be initialized -with retain count 1; its state will be otherwise uninitialized. The -optional ``objc`` attribute indicates that the object should be -allocated using Objective-C's allocation methods (``+allocWithZone:``). -The optional ``stack`` attribute indicates that the object can be allocated -on the stack instead on the heap. In this case the instruction must have -balanced with a ``dealloc_ref [stack]`` instruction to mark the end of the -object's lifetime. -Note that the ``stack`` attribute only specifies that stack allocation is -possible. The final decision on stack allocation is done during llvm IR -generation. This is because the decision also depends on the object size, -which is not necessarily known at SIL level. - -alloc_ref_dynamic -````````````````` -:: - - sil-instruction ::= 'alloc_ref_dynamic' ('[' 'objc' ']')? sil-operand ',' sil-type - - %1 = alloc_ref_dynamic %0 : $@thick T.Type, $T - %1 = alloc_ref_dynamic [objc] %0 : $@objc_metatype T.Type, $T - // $T must be a class type - // %1 has type $T - -Allocates an object of class type ``T`` or a subclass thereof. The -dynamic type of the resulting object is specified via the metatype -value ``%0``. The object will be initialized with retain count 1; its -state will be otherwise uninitialized. The optional ``objc`` attribute -indicates that the object should be allocated using Objective-C's -allocation methods (``+allocWithZone:``). - -alloc_box -````````` -:: - - sil-instruction ::= 'alloc_box' sil-type - - %1 = alloc_box $T - // %1 has two values: - // %1#0 has type $@box T - // %1#1 has type $*T - -Allocates a reference-counted ``@box`` on the heap large enough to hold a value -of type ``T``, along with a retain count and any other metadata required by the -runtime. The result of the instruction is a two-value operand; the first value -is the reference-counted ``@box`` reference that owns the box, and the second -value is the address of the value inside the box. - -The box will be initialized with a retain count of 1; the storage will be -uninitialized. The box owns the contained value, and releasing it to a retain -count of zero destroys the contained value as if by ``destroy_addr``. -Releasing a box is undefined behavior if the box's value is uninitialized. -To deallocate a box whose value has not been initialized, ``dealloc_box`` -should be used. - -alloc_value_buffer -`````````````````` - -:: - - sil-instruction ::= 'alloc_value_buffer' sil-type 'in' sil-operand - - %1 = alloc_value_buffer $(Int, T) in %0 : $*Builtin.UnsafeValueBuffer - // The operand must have the exact type shown. - // The result has type $*(Int, T). - -Given the address of an unallocated value buffer, allocate space in it -for a value of the given type. This instruction has undefined -behavior if the value buffer is currently allocated. - -The type operand must be a lowered object type. - -dealloc_stack -````````````` -:: - - sil-instruction ::= 'dealloc_stack' sil-operand - - dealloc_stack %0 : $*@local_storage T - // %0 must be of a local-storage $*@local_storage T type - -Deallocates memory previously allocated by ``alloc_stack``. The -allocated value in memory must be uninitialized or destroyed prior to -being deallocated. This instruction marks the end of the lifetime for -the value created by the corresponding ``alloc_stack`` instruction. The operand -must be the ``@local_storage`` of the shallowest live ``alloc_stack`` -allocation preceding the deallocation. In other words, deallocations must be -in last-in, first-out stack order. - -dealloc_box -``````````` -:: - - sil-instruction ::= 'dealloc_box' sil-operand - - dealloc_box %0 : $@box T - -Deallocates a box, bypassing the reference counting mechanism. The box -variable must have a retain count of one. The boxed type must match the -type passed to the corresponding ``alloc_box`` exactly, or else -undefined behavior results. - -This does not destroy the boxed value. The contents of the -value must have been fully uninitialized or destroyed before -``dealloc_box`` is applied. - -project_box -``````````` -:: - - sil-instruction ::= 'project_box' sil-operand - - %1 = project_box %0 : $@box T - - // %1 has type $*T - -Given a ``@box T`` reference, produces the address of the value inside the box. - -dealloc_ref -``````````` -:: - - sil-instruction ::= 'dealloc_ref' ('[' 'stack' ']')? sil-operand - - dealloc_ref [stack] %0 : $T - // $T must be a class type - -Deallocates an uninitialized class type instance, bypassing the reference -counting mechanism. - -The type of the operand must match the allocated type exactly, or else -undefined behavior results. - -The instance must have a retain count of one. - -This does not destroy stored properties of the instance. The contents -of stored properties must be fully uninitialized at the time -``dealloc_ref`` is applied. - -The ``stack`` attribute indicates that the instruction is the balanced -deallocation of its operand which must be a ``alloc_ref [stack]``. -In this case the instruction marks the end of the object's lifetime but -has no other effect. - -dealloc_partial_ref -``````````````````` -:: - - sil-instruction ::= 'dealloc_partial_ref' sil-operand sil-metatype - - dealloc_partial_ref %0 : $T, %1 : $U.Type - // $T must be a class type - // $T must be a subclass of U - -Deallocates a partially-initialized class type instance, bypassing -the reference counting mechanism. - -The type of the operand must be a supertype of the allocated type, or -else undefined behavior results. - -The instance must have a retain count of one. - -All stored properties in classes more derived than the given metatype -value must be initialized, and all other stored properties must be -uninitialized. The initialized stored properties are destroyed before -deallocating the memory for the instance. - -This does not destroy the reference type instance. The contents of the -heap object must have been fully uninitialized or destroyed before -``dealloc_ref`` is applied. - -dealloc_value_buffer -```````````````````` - -:: - - sil-instruction ::= 'dealloc_value_buffer' sil-type 'in' sil-operand - - dealloc_value_buffer $(Int, T) in %0 : $*Builtin.UnsafeValueBuffer - // The operand must have the exact type shown. - -Given the address of a value buffer, deallocate the storage in it. -This instruction has undefined behavior if the value buffer is not -currently allocated, or if it was allocated with a type other than the -type operand. - -The type operand must be a lowered object type. - -project_value_buffer -```````````````````` - -:: - - sil-instruction ::= 'project_value_buffer' sil-type 'in' sil-operand - - %1 = project_value_buffer $(Int, T) in %0 : $*Builtin.UnsafeValueBuffer - // The operand must have the exact type shown. - // The result has type $*(Int, T). - -Given the address of a value buffer, return the address of the value -storage in it. This instruction has undefined behavior if the value -buffer is not currently allocated, or if it was allocated with a type -other than the type operand. - -The result is the same value as was originally returned by -``alloc_value_buffer``. - -The type operand must be a lowered object type. - -Debug Information -~~~~~~~~~~~~~~~~~ - -Debug information is generally associated with allocations (alloc_stack or -alloc_box) by having a Decl node attached to the allocation with a SILLocation. -For declarations that have no allocation we have explicit instructions for -doing this. This is used by 'let' declarations, which bind a value to a name -and for var decls who are promoted into registers. The decl they refer to is -attached to the instruction with a SILLocation. - -debug_value -``````````` - -:: - - sil-instruction ::= debug_value sil-operand - - debug_value %1 : $Int - -This indicates that the value of a declaration with loadable type has changed -value to the specified operand. The declaration in question is identified by -the SILLocation attached to the debug_value instruction. - -The operand must have loadable type. - -debug_value_addr -```````````````` - -:: - - sil-instruction ::= debug_value_addr sil-operand - - debug_value_addr %7 : $*SomeProtocol - -This indicates that the value of a declaration with address-only type -has changed value to the specified operand. The declaration in -question is identified by the SILLocation attached to the -debug_value_addr instruction. - - -Accessing Memory -~~~~~~~~~~~~~~~~ - -load -```` -:: - - sil-instruction ::= 'load' sil-operand - - %1 = load %0 : $*T - // %0 must be of a $*T address type for loadable type $T - // %1 will be of type $T - -Loads the value at address ``%0`` from memory. ``T`` must be a loadable type. -This does not affect the reference count, if any, of the loaded value; the -value must be retained explicitly if necessary. It is undefined behavior to -load from uninitialized memory or to load from an address that points to -deallocated storage. - -store -````` -:: - - sil-instruction ::= 'store' sil-value 'to' sil-operand - - store %0 to %1 : $*T - // $T must be a loadable type - -Stores the value ``%0`` to memory at address ``%1``. The type of %1 is ``*T`` -and the type of ``%0 is ``T``, which must be a loadable type. This will -overwrite the memory at ``%1``. If ``%1`` already references a value that -requires ``release`` or other cleanup, that value must be loaded before being -stored over and cleaned up. It is undefined behavior to store to an address -that points to deallocated storage. - -assign -`````` -:: - - sil-instruction ::= 'assign' sil-value 'to' sil-operand - - assign %0 to %1 : $*T - // $T must be a loadable type - -Represents an abstract assignment of the value ``%0`` to memory at address -``%1`` without specifying whether it is an initialization or a normal store. -The type of %1 is ``*T`` and the type of ``%0`` is ``T``, which must be a -loadable type. This will overwrite the memory at ``%1`` and destroy the value -currently held there. - -The purpose of the ``assign`` instruction is to simplify the -definitive initialization analysis on loadable variables by removing -what would otherwise appear to be a load and use of the current value. -It is produced by SILGen, which cannot know which assignments are -meant to be initializations. If it is deemed to be an initialization, -it can be replaced with a ``store``; otherwise, it must be replaced -with a sequence that also correctly destroys the current value. - -This instruction is only valid in Raw SIL and is rewritten as appropriate -by the definitive initialization pass. - -mark_uninitialized -`````````````````` -:: - - sil-instruction ::= 'mark_uninitialized' '[' mu_kind ']' sil-operand - mu_kind ::= 'var' - mu_kind ::= 'rootself' - mu_kind ::= 'derivedself' - mu_kind ::= 'derivedselfonly' - mu_kind ::= 'delegatingself' - - %2 = mark_uninitialized [var] %1 : $*T - // $T must be an address - -Indicates that a symbolic memory location is uninitialized, and must be -explicitly initialized before it escapes or before the current function returns. -This instruction returns its operands, and all accesses within the function must -be performed against the return value of the mark_uninitialized instruction. - -The kind of mark_uninitialized instruction specifies the type of data -the mark_uninitialized instruction refers to: - -- ``var``: designates the start of a normal variable live range -- ``rootself``: designates ``self`` in a struct, enum, or root class -- ``derivedself``: designates ``self`` in a derived (non-root) class -- ``derivedselfonly``: designates ``self`` in a derived (non-root) class whose stored properties have already been initialized -- ``delegatingself``: designates ``self`` on a struct, enum, or class in a delegating constructor (one that calls self.init) - -The purpose of the ``mark_uninitialized`` instruction is to enable -definitive initialization analysis for global variables (when marked as -'globalvar') and instance variables (when marked as 'rootinit'), which need to -be distinguished from simple allocations. - -It is produced by SILGen, and is only valid in Raw SIL. It is rewritten as -appropriate by the definitive initialization pass. - -mark_function_escape -```````````````````` -:: - - sil-instruction ::= 'mark_function_escape' sil-operand (',' sil-operand) - - %2 = mark_function_escape %1 : $*T - -Indicates that a function definition closes over a symbolic memory location. -This instruction is variadic, and all of its operands must be addresses. - -The purpose of the ``mark_function_escape`` instruction is to enable -definitive initialization analysis for global variables and instance variables, -which are not represented as box allocations. - -It is produced by SILGen, and is only valid in Raw SIL. It is rewritten as -appropriate by the definitive initialization pass. - -copy_addr -````````` -:: - - sil-instruction ::= 'copy_addr' '[take]'? sil-value - 'to' '[initialization]'? sil-operand - - copy_addr [take] %0 to [initialization] %1 : $*T - // %0 and %1 must be of the same $*T address type - -Loads the value at address ``%0`` from memory and assigns a copy of it back into -memory at address ``%1``. A bare ``copy_addr`` instruction when ``T`` is a -non-trivial type:: - - copy_addr %0 to %1 : $*T - -is equivalent to:: - - %new = load %0 : $*T // Load the new value from the source - %old = load %1 : $*T // Load the old value from the destination - strong_retain %new : $T // Retain the new value - strong_release %old : $T // Release the old - store %new to %1 : $*T // Store the new value to the destination - -except that ``copy_addr`` may be used even if ``%0`` is of an address-only -type. The ``copy_addr`` may be given one or both of the ``[take]`` or -``[initialization]`` attributes: - -* ``[take]`` destroys the value at the source address in the course of the - copy. -* ``[initialization]`` indicates that the destination address is uninitialized. - Without the attribute, the destination address is treated as already - initialized, and the existing value will be destroyed before the new value - is stored. - -The three attributed forms thus behave like the following loadable type -operations:: - - // take-assignment - copy_addr [take] %0 to %1 : $*T - // is equivalent to: - %new = load %0 : $*T - %old = load %1 : $*T - // no retain of %new! - strong_release %old : $T - store %new to %1 : $*T - - // copy-initialization - copy_addr %0 to [initialization] %1 : $*T - // is equivalent to: - %new = load %0 : $*T - strong_retain %new : $T - // no load/release of %old! - store %new to %1 : $*T - - // take-initialization - copy_addr [take] %0 to [initialization] %1 : $*T - // is equivalent to: - %new = load %0 : $*T - // no retain of %new! - // no load/release of %old! - store %new to %1 : $*T - -If ``T`` is a trivial type, then ``copy_addr`` is always equivalent to its -take-initialization form. - -destroy_addr -```````````` -:: - - sil-instruction ::= 'destroy_addr' sil-operand - - destroy_addr %0 : $*T - // %0 must be of an address $*T type - -Destroys the value in memory at address ``%0``. If ``T`` is a non-trivial type, -This is equivalent to:: - - %1 = load %0 - strong_release %1 - -except that ``destroy_addr`` may be used even if ``%0`` is of an -address-only type. This does not deallocate memory; it only destroys the -pointed-to value, leaving the memory uninitialized. - -If ``T`` is a trivial type, then ``destroy_addr`` is a no-op. - -index_addr -`````````` -:: - - sil-instruction ::= 'index_addr' sil-operand ',' sil-operand - - %2 = index_addr %0 : $*T, %1 : $Builtin.Int - // %0 must be of an address type $*T - // %1 must be of a builtin integer type - // %2 will be of type $*T - -Given an address that references into an array of values, returns the address -of the ``%1``-th element relative to ``%0``. The address must reference into -a contiguous array. It is undefined to try to reference offsets within a -non-array value, such as fields within a homogeneous struct or tuple type, or -bytes within a value, using ``index_addr``. (``Int8`` address types have no -special behavior in this regard, unlike ``char*`` or ``void*`` in C.) It is -also undefined behavior to index out of bounds of an array, except to index -the "past-the-end" address of the array. - -index_raw_pointer -````````````````` -:: - - sil-instruction ::= 'index_raw_pointer' sil-operand ',' sil-operand - - %2 = index_raw_pointer %0 : $Builtin.RawPointer, %1 : $Builtin.Int - // %0 must be of $Builtin.RawPointer type - // %1 must be of a builtin integer type - // %2 will be of type $*T - -Given a ``Builtin.RawPointer`` value ``%0``, returns a pointer value at the -byte offset ``%1`` relative to ``%0``. - -Reference Counting -~~~~~~~~~~~~~~~~~~ - -These instructions handle reference counting of heap objects. Values of -strong reference type have ownership semantics for the referenced heap -object. Retain and release operations, however, -are never implicit in SIL and always must be explicitly performed where needed. -Retains and releases on the value may be freely moved, and balancing -retains and releases may deleted, so long as an owning retain count is -maintained for the uses of the value. - -All reference-counting operations are defined to work correctly on -null references (whether strong, unowned, or weak). A non-null -reference must actually refer to a valid object of the indicated type -(or a subtype). Address operands are required to be valid and non-null. - -While SIL makes reference-counting operations explicit, the SIL type -system also fully represents strength of reference. This is useful -for several reasons: - -1. Type-safety: it is impossible to erroneously emit SIL that naively - uses a ``@weak`` or ``@unowned`` reference as if it were a strong - reference. - -2. Consistency: when a reference is kept in memory, instructions like - ``copy_addr`` and ``destroy_addr`` implicitly carry the right - semantics in the type of the address, rather than needing special - variants or flags. - -3. Ease of tooling: SIL directly stores the user's intended strength - of reference, making it straightforward to generate instrumentation - that would convey this to a memory profiler. In principle, with - only a modest number of additions and restrictions on SIL, it would - even be possible to drop all reference-counting instructions and - use the type information to feed a garbage collector. - -strong_retain -````````````` -:: - - sil-instruction ::= 'strong_retain' sil-operand - - strong_retain %0 : $T - // $T must be a reference type - -Increases the strong retain count of the heap object referenced by ``%0``. - -strong_retain_autoreleased -`````````````````````````` -:: - - sil-instruction ::= 'strong_retain_autoreleased' sil-operand - - strong_retain_autoreleased %0 : $T - // $T must have a retainable pointer representation - -Retains the heap object referenced by ``%0`` using the Objective-C ARC -"autoreleased return value" optimization. The operand must be the result of an -``apply`` instruction with an Objective-C method callee, and the -``strong_retain_autoreleased`` instruction must be first use of the value after -the defining ``apply`` instruction. - -TODO: Specify all the other strong_retain_autoreleased constraints here. - -strong_release -`````````````` -:: - - strong_release %0 : $T - // $T must be a reference type. - -Decrements the strong reference count of the heap object referenced by ``%0``. -If the release operation brings the strong reference count of the object to -zero, the object is destroyed and ``@weak`` references are cleared. When both -its strong and unowned reference counts reach zero, the object's memory is -deallocated. - -strong_retain_unowned -````````````````````` -:: - - sil-instruction ::= 'strong_retain_unowned' sil-operand - - strong_retain_unowned %0 : $@unowned T - // $T must be a reference type - -Asserts that the strong reference count of the heap object referenced by ``%0`` -is still positive, then increases it by one. - -unowned_retain -`````````````` -:: - - sil-instruction ::= 'unowned_retain' sil-operand - - unowned_retain %0 : $@unowned T - // $T must be a reference type - -Increments the unowned reference count of the heap object underlying ``%0``. - -unowned_release -``````````````` -:: - - sil-instruction ::= 'unowned_release' sil-operand - - unowned_release %0 : $@unowned T - // $T must be a reference type - -Decrements the unowned reference count of the heap object referenced by -``%0``. When both its strong and unowned reference counts reach zero, -the object's memory is deallocated. - -load_weak -````````` - -:: - - sil-instruction ::= 'load_weak' '[take]'? sil-operand - - load_weak [take] %0 : $*@sil_weak Optional - // $T must be an optional wrapping a reference type - -Increments the strong reference count of the heap object held in the operand, -which must be an initialized weak reference. The result is value of type -``$Optional``, except that it is ``null`` if the heap object has begun -deallocation. - -This operation must be atomic with respect to the final ``strong_release`` on -the operand heap object. It need not be atomic with respect to ``store_weak`` -operations on the same address. - -store_weak -`````````` - -:: - - sil-instruction ::= 'store_weak' sil-value 'to' '[initialization]'? sil-operand - - store_weak %0 to [initialization] %1 : $*@sil_weak Optional - // $T must be an optional wrapping a reference type - -Initializes or reassigns a weak reference. The operand may be ``nil``. - -If ``[initialization]`` is given, the weak reference must currently either be -uninitialized or destroyed. If it is not given, the weak reference must -currently be initialized. - -This operation must be atomic with respect to the final ``strong_release`` on -the operand (source) heap object. It need not be atomic with respect to -``store_weak`` or ``load_weak`` operations on the same address. - -fix_lifetime -```````````` - -:: - - sil-instruction :: 'fix_lifetime' sil-operand - - fix_lifetime %0 : $T - // Fix the lifetime of a value %0 - fix_lifetime %1 : $*T - // Fix the lifetime of the memory object referenced by %1 - -Acts as a use of a value operand, or of the value in memory referenced by an -address operand. Optimizations may not move operations that would destroy the -value, such as ``release_value``, ``strong_release``, ``copy_addr [take]``, or -``destroy_addr``, past this instruction. - -mark_dependence -``````````````` - -:: - - sil-instruction :: 'mark_dependence' sil-operand 'on' sil-operand - - %2 = mark_dependence %0 : $*T on %1 : $Builtin.NativeObject - -Indicates that the validity of the first operand depends on the value -of the second operand. Operations that would destroy the second value -must not be moved before any instructions which depend on the result -of this instruction, exactly as if the address had been obviously -derived from that operand (e.g. using ``ref_element_addr``). - -The result is always equal to the first operand. The first operand -will typically be an address, but it could be an address in a -non-obvious form, such as a Builtin.RawPointer or a struct containing -the same. Transformations should be somewhat forgiving here. - -The second operand may have either object or address type. In the -latter case, the dependency is on the current value stored in the -address. - -is_unique -````````` - -:: - - sil-instruction ::= 'is_unique' sil-operand - - %1 = is_unique %0 : $*T - // $T must be a reference-counted type - // %1 will be of type Builtin.Int1 - -Checks whether %0 is the address of a unique reference to a memory -object. Returns 1 if the strong reference count is 1, and 0 if the -strong reference count is greater than 1. - -A discussion of the semantics can be found here: -:ref:`arcopts.is_unique`. - -is_unique_or_pinned -``````````````````` - -:: - - sil-instruction ::= 'is_unique_or_pinned' sil-operand - - %1 = is_unique_or_pinned %0 : $*T - // $T must be a reference-counted type - // %1 will be of type Builtin.Int1 - -Checks whether %0 is the address of either a unique reference to a -memory object or a reference to a pinned object. Returns 1 if the -strong reference count is 1 or the object has been marked pinned by -strong_pin. - -copy_block -`````````` -:: - - sil-instruction :: 'copy_block' sil-operand - - %1 = copy_block %0 : $@convention(block) T -> U - -Performs a copy of an Objective-C block. Unlike retains of other -reference-counted types, this can produce a different value from the operand -if the block is copied from the stack to the heap. - -Literals -~~~~~~~~ - -These instructions bind SIL values to literal constants or to global entities. - -function_ref -```````````` -:: - - sil-instruction ::= 'function_ref' sil-function-name ':' sil-type - - %1 = function_ref @function : $@thin T -> U - // $@thin T -> U must be a thin function type - // %1 has type $T -> U - -Creates a reference to a SIL function. - -global_addr -``````````````` - -:: - - sil-instruction ::= 'global_addr' sil-global-name ':' sil-type - - %1 = global_addr @foo : $*Builtin.Word - -Creates a reference to the address of a global variable. - -integer_literal -``````````````` -:: - - sil-instruction ::= 'integer_literal' sil-type ',' int-literal - - %1 = integer_literal $Builtin.Int, 123 - // $Builtin.Int must be a builtin integer type - // %1 has type $Builtin.Int - -Creates an integer literal value. The result will be of type -``Builtin.Int``, which must be a builtin integer type. The literal value -is specified using Swift's integer literal syntax. - -float_literal -````````````` -:: - - sil-instruction ::= 'float_literal' sil-type ',' int-literal - - %1 = float_literal $Builtin.FP, 0x3F800000 - // $Builtin.FP must be a builtin floating-point type - // %1 has type $Builtin.FP - -Creates a floating-point literal value. The result will be of type `` -``Builtin.FP``, which must be a builtin floating-point type. The literal -value is specified as the bitwise representation of the floating point value, -using Swift's hexadecimal integer literal syntax. - -string_literal -`````````````` -:: - - sil-instruction ::= 'string_literal' encoding string-literal - encoding ::= 'utf8' - encoding ::= 'utf16' - - %1 = string_literal "asdf" - // %1 has type $Builtin.RawPointer - -Creates a reference to a string in the global string table. The result -is a pointer to the data. The referenced string is always null-terminated. The -string literal value is specified using Swift's string -literal syntax (though ``\()`` interpolations are not allowed). - -Dynamic Dispatch -~~~~~~~~~~~~~~~~ - -These instructions perform dynamic lookup of class and generic methods. They -share a common set of attributes:: - - sil-method-attributes ::= '[' 'volatile'? ']' - -The ``volatile`` attribute on a dynamic dispatch instruction indicates that -the method lookup is semantically required (as, for example, in Objective-C). -When the type of a dynamic dispatch instruction's operand is known, -optimization passes can promote non-``volatile`` dispatch instructions -into static ``function_ref`` instructions. - -If a dynamic dispatch instruction references an Objective-C method -(indicated by the ``foreign`` marker on a method reference, as in -``#NSObject.description!1.foreign``), then the instruction -represents an ``objc_msgSend`` invocation. ``objc_msgSend`` invocations can -only be used as the callee of an ``apply`` instruction or ``partial_apply`` -instruction. They cannot be stored or used as ``apply`` or ``partial_apply`` -arguments. ``objc_msgSend`` invocations must always be ``volatile``. - -class_method -```````````` -:: - - sil-instruction ::= 'class_method' sil-method-attributes? - sil-operand ',' sil-decl-ref ':' sil-type - - %1 = class_method %0 : $T, #T.method!1 : $@thin U -> V - // %0 must be of a class type or class metatype $T - // #T.method!1 must be a reference to a dynamically-dispatched method of T or - // of one of its superclasses, at uncurry level >= 1 - // %1 will be of type $U -> V - -Looks up a method based on the dynamic type of a class or class metatype -instance. It is undefined behavior if the class value is null and the -method is not an Objective-C method. - -If: - -- the instruction is not ``[volatile]``, -- the referenced method is not a ``foreign`` method, -- and the static type of the class instance is known, or the method is known - to be final, - -then the instruction is a candidate for devirtualization optimization. A -devirtualization pass can consult the module's `VTables`_ to find the -SIL function that implements the method and promote the instruction to a -static `function_ref`_. - -super_method -```````````` -:: - - sil-instruction ::= 'super_method' sil-method-attributes? - sil-operand ',' sil-decl-ref ':' sil-type - - %1 = super_method %0 : $T, #Super.method!1.foreign : $@thin U -> V - // %0 must be of a non-root class type or class metatype $T - // #Super.method!1.foreign must be a reference to an ObjC method of T's - // superclass or of one of its ancestor classes, at uncurry level >= 1 - // %1 will be of type $@thin U -> V - -Looks up a method in the superclass of a class or class metatype instance. -Note that for native Swift methods, ``super.method`` calls are statically -dispatched, so this instruction is only valid for Objective-C methods. -It is undefined behavior if the class value is null and the method is -not an Objective-C method. - -witness_method -`````````````` -:: - - sil-instruction ::= 'witness_method' sil-method-attributes? - sil-type ',' sil-decl-ref ':' sil-type - - %1 = witness_method $T, #Proto.method!1 \ - : $@thin @cc(witness_method) U -> V - // $T must be an archetype - // #Proto.method!1 must be a reference to a method of one of the protocol - // constraints on T - // U -> V must be the type of the referenced method, - // generic on Self - // %1 will be of type $@thin U -> V - -Looks up the implementation of a protocol method for a generic type variable -constrained by that protocol. The result will be generic on the ``Self`` -archetype of the original protocol and have the ``witness_method`` calling -convention. If the referenced protocol is an ``@objc`` protocol, the -resulting type has the ``objc`` calling convention. - -dynamic_method -`````````````` -:: - - sil-instruction ::= 'dynamic_method' sil-method-attributes? - sil-operand ',' sil-decl-ref ':' sil-type - - %1 = dynamic_method %0 : $P, #X.method!1 : $@thin U -> V - // %0 must be of a protocol or protocol composition type $P, - // where $P contains the Swift.DynamicLookup protocol - // #X.method!1 must be a reference to an @objc method of any class - // or protocol type - // - // The "self" argument of the method type $@thin U -> V must be - // Builtin.ObjCPointer - -Looks up the implementation of an Objective-C method with the same -selector as the named method for the dynamic type of the -value inside an existential container. The "self" operand of the result -function value is represented using an opaque type, the value for which must -be projected out as a value of type ``Builtin.ObjCPointer``. - -It is undefined behavior if the dynamic type of the operand does not -have an implementation for the Objective-C method with the selector to -which the ``dynamic_method`` instruction refers, or if that -implementation has parameter or result types that are incompatible -with the method referenced by ``dynamic_method``. -This instruction should only be used in cases where its result will be -immediately consumed by an operation that performs the selector check -itself (e.g., an ``apply`` that lowers to ``objc_msgSend``). -To query whether the operand has an implementation for the given -method and safely handle the case where it does not, use -`dynamic_method_br`_. - -Function Application -~~~~~~~~~~~~~~~~~~~~ - -These instructions call functions or wrap them in partial application or -specialization thunks. - -apply -````` -:: - - sil-instruction ::= 'apply' '[nothrow]'? sil-value - sil-apply-substitution-list? - '(' (sil-value (',' sil-value)*)? ')' - ':' sil-type - - sil-apply-substitution-list ::= '<' sil-substitution - (',' sil-substitution)* '>' - sil-substitution ::= type '=' type - - %r = apply %0(%1, %2, ...) : $(A, B, ...) -> R - // Note that the type of the callee '%0' is specified *after* the arguments - // %0 must be of a concrete function type $(A, B, ...) -> R - // %1, %2, etc. must be of the argument types $A, $B, etc. - // %r will be of the return type $R - - %r = apply %0(%1, %2, ...) : $(T, U, ...) -> R - // %0 must be of a polymorphic function type $(T, U, ...) -> R - // %1, %2, etc. must be of the argument types after substitution $A, $B, etc. - // %r will be of the substituted return type $R' - -Transfers control to function ``%0``, passing it the given arguments. In -the instruction syntax, the type of the callee is specified after the argument -list; the types of the argument and of the defined value are derived from the -function type of the callee. The input argument tuple type is destructured, -and each element is passed as an individual argument. The ``apply`` -instruction does no retaining or releasing of its arguments by itself; the -`calling convention`_'s retain/release policy must be handled by separate -explicit ``retain`` and ``release`` instructions. The return value will -likewise not be implicitly retained or released. - -The callee value must have function type. That function type may not -have an error result, except the instruction has the ``nothrow`` attribute set. -The ``nothrow`` attribute specifies that the callee has an error result but -does not actually throw. -For the regular case of calling a function with error result, use ``try_apply``. - -NB: If the callee value is of a thick function type, ``apply`` currently -consumes the callee value at +1 strong retain count. - -If the callee is generic, all of its generic parameters must be bound by the -given substitution list. The arguments and return value is -given with these generic substitutions applied. - -partial_apply -````````````` -:: - - sil-instruction ::= 'partial_apply' sil-value - sil-apply-substitution-list? - '(' (sil-value (',' sil-value)*)? ')' - ':' sil-type - - %c = partial_apply %0(%1, %2, ...) : $(Z..., A, B, ...) -> R - // Note that the type of the callee '%0' is specified *after* the arguments - // %0 must be of a concrete function type $(Z..., A, B, ...) -> R - // %1, %2, etc. must be of the argument types $A, $B, etc., - // of the tail part of the argument tuple of %0 - // %c will be of the partially-applied thick function type (Z...) -> R - - %c = partial_apply %0(%1, %2, ...) : $(Z..., T, U, ...) -> R - // %0 must be of a polymorphic function type $(T, U, ...) -> R - // %1, %2, etc. must be of the argument types after substitution $A, $B, etc. - // of the tail part of the argument tuple of %0 - // %r will be of the substituted thick function type $(Z'...) -> R' - -Creates a closure by partially applying the function ``%0`` to a partial -sequence of its arguments. In the instruction syntax, the type of the callee is -specified after the argument list; the types of the argument and of the defined -value are derived from the function type of the callee. The closure context will -be allocated with retain count 1 and initialized to contain the values ``%1``, -``%2``, etc. The closed-over values will not be retained; that must be done -separately before the ``partial_apply``. The closure does however take -ownership of the partially applied arguments; when the closure reference -count reaches zero, the contained values will be destroyed. - -If the callee is generic, all of its generic parameters must be bound by the -given substitution list. The arguments are given with these generic -substitutions applied, and the resulting closure is of concrete function -type with the given substitutions applied. The generic parameters themselves -cannot be partially applied; all of them must be bound. The result is always -a concrete function. - -TODO: The instruction, when applied to a generic function, -currently implicitly performs abstraction difference transformations enabled -by the given substitutions, such as promoting address-only arguments and returns -to register arguments. This should be fixed. - -This instruction is used to implement both curry thunks and closures. A -curried function in Swift:: - - func foo(a:A)(b:B)(c:C)(d:D) -> E { /* body of foo */ } - -emits curry thunks in SIL as follows (retains and releases omitted for -clarity):: - - func @foo : $@thin A -> B -> C -> D -> E { - entry(%a : $A): - %foo_1 = function_ref @foo_1 : $@thin (B, A) -> C -> D -> E - %thunk = partial_apply %foo_1(%a) : $@thin (B, A) -> C -> D -> E - return %thunk : $B -> C -> D -> E - } - - func @foo_1 : $@thin (B, A) -> C -> D -> E { - entry(%b : $B, %a : $A): - %foo_2 = function_ref @foo_2 : $@thin (C, B, A) -> D -> E - %thunk = partial_apply %foo_2(%b, %a) : $@thin (C, B, A) -> D -> E - return %thunk : $(B, A) -> C -> D -> E - } - - func @foo_2 : $@thin (C, B, A) -> D -> E { - entry(%c : $C, %b : $B, %a : $A): - %foo_3 = function_ref @foo_3 : $@thin (D, C, B, A) -> E - %thunk = partial_apply %foo_3(%c, %b, %a) : $@thin (D, C, B, A) -> E - return %thunk : $(C, B, A) -> D -> E - } - - func @foo_3 : $@thin (D, C, B, A) -> E { - entry(%d : $D, %c : $C, %b : $B, %a : $A): - // ... body of foo ... - } - -A local function in Swift that captures context, such as ``bar`` in the -following example:: - - func foo(x:Int) -> Int { - func bar(y:Int) -> Int { - return x + y - } - return bar(1) - } - -lowers to an uncurried entry point and is curried in the enclosing function:: - - func @bar : $@thin (Int, @box Int, *Int) -> Int { - entry(%y : $Int, %x_box : $@box Int, %x_address : $*Int): - // ... body of bar ... - } - - func @foo : $@thin Int -> Int { - entry(%x : $Int): - // Create a box for the 'x' variable - %x_box = alloc_box $Int - store %x to %x_box#1 : $*Int - - // Create the bar closure - %bar_uncurried = function_ref @bar : $(Int, Int) -> Int - %bar = partial_apply %bar_uncurried(%x_box#0, %x_box#1) \ - : $(Int, Builtin.ObjectPointer, *Int) -> Int - - // Apply it - %1 = integer_literal $Int, 1 - %ret = apply %bar(%1) : $(Int) -> Int - - // Clean up - release %bar : $(Int) -> Int - return %ret : $Int - } - -builtin -``````` -:: - - sil-instruction ::= 'builtin' string-literal - sil-apply-substitution-list? - '(' (sil-operand (',' sil-operand)*)? ')' - ':' sil-type - - %1 = builtin "foo"(%1 : $T, %2 : $U) : $V - // "foo" must name a function in the Builtin module - -Invokes functionality built into the backend code generator, such as LLVM- -level instructions and intrinsics. - -Metatypes -~~~~~~~~~ - -These instructions access metatypes, either statically by type name or -dynamically by introspecting class or generic values. - -metatype -```````` -:: - - sil-instruction ::= 'metatype' sil-type - - %1 = metatype $T.metatype - // %1 has type $T.metatype - -Creates a reference to the metatype object for type ``T``. - -value_metatype -`````````````` -:: - - sil-instruction ::= 'value_metatype' sil-type ',' sil-operand - - %1 = value_metatype $T.metatype, %0 : $T - // %0 must be a value or address of type $T - // %1 will be of type $T.metatype - -Obtains a reference to the dynamic metatype of the value ``%0``. - -existential_metatype -```````````````````` -:: - - sil-instruction ::= 'existential_metatype' sil-type ',' sil-operand - - %1 = existential_metatype $P.metatype, %0 : $P - // %0 must be a value of class protocol or protocol composition - // type $P, or an address of address-only protocol type $*P - // %1 will be a $P.metatype value referencing the metatype of the - // concrete value inside %0 - -Obtains the metatype of the concrete value -referenced by the existential container referenced by ``%0``. - -objc_protocol -````````````` -:: - - sil-instruction ::= 'objc_protocol' protocol-decl : sil-type - - %0 = objc_protocol #ObjCProto : $Protocol - -*TODO* Fill this in. - -Aggregate Types -~~~~~~~~~~~~~~~ - -These instructions construct and project elements from structs, tuples, and -class instances. - -retain_value -```````````` - -:: - - sil-instruction ::= 'retain_value' sil-operand - - retain_value %0 : $A - -Retains a loadable value, which simply retains any references it holds. - -For trivial types, this is a no-op. For reference types, this is equivalent to -a ``strong_retain``. For ``@unowned`` types, this is equivalent to an -``unowned_retain``. In each of these cases, those are the preferred forms. - -For aggregate types, especially enums, it is typically both easier -and more efficient to reason about aggregate copies than it is to -reason about copies of the subobjects. - -release_value -````````````` - -:: - - sil-instruction ::= 'release_value' sil-operand - - release_value %0 : $A - -Destroys a loadable value, by releasing any retainable pointers within it. - -This is defined to be equivalent to storing the operand into a stack -allocation and using 'destroy_addr' to destroy the object there. - -For trivial types, this is a no-op. For reference types, this is -equivalent to a ``strong_release``. For ``@unowned`` types, this is -equivalent to an ``unowned_release``. In each of these cases, those -are the preferred forms. - -For aggregate types, especially enums, it is typically both easier -and more efficient to reason about aggregate destroys than it is to -reason about destroys of the subobjects. - -autorelease_value -````````````````` - -:: - - sil-instruction ::= 'autorelease_value' sil-operand - - autorelease_value %0 : $A - -*TODO* Complete this section. - -tuple -````` -:: - - sil-instruction ::= 'tuple' sil-tuple-elements - sil-tuple-elements ::= '(' (sil-operand (',' sil-operand)*)? ')' - sil-tuple-elements ::= sil-type '(' (sil-value (',' sil-value)*)? ')' - - %1 = tuple (%a : $A, %b : $B, ...) - // $A, $B, etc. must be loadable non-address types - // %1 will be of the "simple" tuple type $(A, B, ...) - - %1 = tuple $(a:A, b:B, ...) (%a, %b, ...) - // (a:A, b:B, ...) must be a loadable tuple type - // %1 will be of the type $(a:A, b:B, ...) - -Creates a loadable tuple value by aggregating multiple loadable values. - -If the destination type is a "simple" tuple type, that is, it has no keyword -argument labels or variadic arguments, then the first notation can be used, -which interleaves the element values and types. If keyword names or variadic -fields are specified, then the second notation must be used, which spells out -the tuple type before the fields. - -tuple_extract -````````````` -:: - - sil-instruction ::= 'tuple_extract' sil-operand ',' int-literal - - %1 = tuple_extract %0 : $(T...), 123 - // %0 must be of a loadable tuple type $(T...) - // %1 will be of the type of the selected element of %0 - -Extracts an element from a loadable tuple value. - -tuple_element_addr -`````````````````` -:: - - sil-instruction ::= 'tuple_element_addr' sil-operand ',' int-literal - - %1 = tuple_element_addr %0 : $*(T...), 123 - // %0 must of a $*(T...) address-of-tuple type - // %1 will be of address type $*U where U is the type of the 123rd - // element of T - -Given the address of a tuple in memory, derives the -address of an element within that value. - -struct -`````` -:: - - sil-instruction ::= 'struct' sil-type '(' (sil-operand (',' sil-operand)*)? ')' - - %1 = struct $S (%a : $A, %b : $B, ...) - // $S must be a loadable struct type - // $A, $B, ... must be the types of the physical 'var' fields of $S in order - // %1 will be of type $S - -Creates a value of a loadable struct type by aggregating multiple loadable -values. - -struct_extract -`````````````` -:: - - sil-instruction ::= 'struct_extract' sil-operand ',' sil-decl-ref - - %1 = struct_extract %0 : $S, #S.field - // %0 must be of a loadable struct type $S - // #S.field must be a physical 'var' field of $S - // %1 will be of the type of the selected field of %0 - -Extracts a physical field from a loadable struct value. - -struct_element_addr -``````````````````` -:: - - sil-instruction ::= 'struct_element_addr' sil-operand ',' sil-decl-ref - - %1 = struct_element_addr %0 : $*S, #S.field - // %0 must be of a struct type $S - // #S.field must be a physical 'var' field of $S - // %1 will be the address of the selected field of %0 - -Given the address of a struct value in memory, derives the address of a -physical field within the value. - -ref_element_addr -```````````````` -:: - - sil-instruction ::= 'ref_element_addr' sil-operand ',' sil-decl-ref - - %1 = ref_element_addr %0 : $C, #C.field - // %0 must be a value of class type $C - // #C.field must be a non-static physical field of $C - // %1 will be of type $*U where U is the type of the selected field - // of C - -Given an instance of a class, derives the address of a physical instance -variable inside the instance. It is undefined behavior if the class value -is null. - -Enums -~~~~~ - -These instructions construct values of enum type. Loadable enum values are -created with the `enum`_ instruction. Address-only enums require two-step -initialization. First, if the case requires data, that data is stored into -the enum at the address projected by `init_enum_data_addr`_. This step is -skipped for cases without data. Finally, the tag for -the enum is injected with an `inject_enum_addr`_ instruction:: - - enum AddressOnlyEnum { - case HasData(AddressOnlyType) - case NoData - } - - sil @init_with_data : $(AddressOnlyType) -> AddressOnlyEnum { - entry(%0 : $*AddressOnlyEnum, %1 : $*AddressOnlyType): - // Store the data argument for the case. - %2 = init_enum_data_addr %0 : $*AddressOnlyEnum, #AddressOnlyEnum.HasData - copy_addr [take] %2 to [initialization] %1 : $*AddressOnlyType - // Inject the tag. - inject_enum_addr %0 : $*AddressOnlyEnum, #AddressOnlyEnum.HasData - return - } - - sil @init_without_data : $() -> AddressOnlyEnum { - // No data. We only need to inject the tag. - inject_enum_addr %0 : $*AddressOnlyEnum, #AddressOnlyEnum.NoData - return - } - -Accessing the value of a loadable enum is inseparable from dispatching on its -discriminator and is done with the `switch_enum`_ terminator:: - - enum Foo { case A(Int), B(String) } - - sil @switch_foo : $(Foo) -> () { - entry(%foo : $Foo): - switch_enum %foo : $Foo, case #Foo.A: a_dest, case #Foo.B: b_dest - - a_dest(%a : $Int): - /* use %a */ - - b_dest(%b : $String): - /* use %b */ - } - -An address-only enum can be tested by branching on it using the -`switch_enum_addr`_ terminator. Its value can then be taken by destructively -projecting the enum value with `unchecked_take_enum_data_addr`_:: - - enum Foo { case A(T), B(String) } - - sil @switch_foo : $ (Foo) -> () { - entry(%foo : $*Foo): - switch_enum_addr %foo : $*Foo, case #Foo.A: a_dest, case #Foo.B: b_dest - - a_dest: - %a = unchecked_take_enum_data_addr %foo : $*Foo, #Foo.A - /* use %a */ - - b_dest: - %b = unchecked_take_enum_data_addr %foo : $*Foo, #Foo.B - /* use %b */ - } - -enum -```` -:: - - sil-instruction ::= 'enum' sil-type ',' sil-decl-ref (',' sil-operand)? - - %1 = enum $U, #U.EmptyCase - %1 = enum $U, #U.DataCase, %0 : $T - // $U must be an enum type - // #U.DataCase or #U.EmptyCase must be a case of enum $U - // If #U.Case has a data type $T, %0 must be a value of type $T - // If #U.Case has no data type, the operand must be omitted - // %1 will be of type $U - -Creates a loadable enum value in the given ``case``. If the ``case`` has a -data type, the enum value will contain the operand value. - -unchecked_enum_data -``````````````````` -:: - - sil-instruction ::= 'unchecked_enum_data' sil-operand ',' sil-decl-ref - - %1 = unchecked_enum_data %0 : $U, #U.DataCase - // $U must be an enum type - // #U.DataCase must be a case of enum $U with data - // %1 will be of object type $T for the data type of case U.DataCase - -Unsafely extracts the payload data for an enum ``case`` from an enum value. -It is undefined behavior if the enum does not contain a value of the given -case. - -init_enum_data_addr -``````````````````` -:: - - sil-instruction ::= 'init_enum_data_addr' sil-operand ',' sil-decl-ref - - %1 = init_enum_data_addr %0 : $*U, #U.DataCase - // $U must be an enum type - // #U.DataCase must be a case of enum $U with data - // %1 will be of address type $*T for the data type of case U.DataCase - -Projects the address of the data for an enum ``case`` inside an enum. This -does not modify the enum or check its value. It is intended to be used as -part of the initialization sequence for an address-only enum. Storing to -the ``init_enum_data_addr`` for a case followed by ``inject_enum_addr`` with that -same case is guaranteed to result in a fully-initialized enum value of that -case being stored. Loading from the ``init_enum_data_addr`` of an initialized -enum value or injecting a mismatched case tag is undefined behavior. - -The address is invalidated as soon as the operand enum is fully initialized by -an ``inject_enum_addr``. - -inject_enum_addr -```````````````` -:: - - sil-instruction ::= 'inject_enum_addr' sil-operand ',' sil-decl-ref - - inject_enum_addr %0 : $*U, #U.Case - // $U must be an enum type - // #U.Case must be a case of enum $U - // %0 will be overlaid with the tag for #U.Case - -Initializes the enum value referenced by the given address by overlaying the -tag for the given case. If the case has no data, this instruction is sufficient -to initialize the enum value. If the case has data, the data must be stored -into the enum at the ``init_enum_data_addr`` address for the case *before* -``inject_enum_addr`` is applied. It is undefined behavior if -``inject_enum_addr`` is applied for a case with data to an uninitialized enum, -or if ``inject_enum_addr`` is applied for a case with data when data for a -mismatched case has been stored to the enum. - -unchecked_take_enum_data_addr -````````````````````````````` -:: - - sil-instruction ::= 'unchecked_take_enum_data_addr' sil-operand ',' sil-decl-ref - - %1 = unchecked_take_enum_data_addr %0 : $*U, #U.DataCase - // $U must be an enum type - // #U.DataCase must be a case of enum $U with data - // %1 will be of address type $*T for the data type of case U.DataCase - -Invalidates an enum value, and takes the address of the payload for the given -enum ``case`` in-place in memory. The referenced enum value is no longer valid, -but the payload value referenced by the result address is valid and must be -destroyed. It is undefined behavior if the referenced enum does not contain a -value of the given ``case``. The result shares memory with the original enum -value; the enum memory cannot be reinitialized as an enum until the payload has -also been invalidated. - -(1.0 only) - -For the first payloaded case of an enum, ``unchecked_take_enum_data_addr`` -is guaranteed to have no side effects; the enum value will not be invalidated. - -select_enum -``````````` -:: - - sil-instruction ::= 'select_enum' sil-operand sil-select-case* - (',' 'default' sil-value)? - ':' sil-type - - %n = select_enum %0 : $U, \ - case #U.Case1: %1, \ - case #U.Case2: %2, /* ... */ \ - default %3 : $T - - // $U must be an enum type - // #U.Case1, Case2, etc. must be cases of enum $U - // %1, %2, %3, etc. must have type $T - // %n has type $T - -Selects one of the "case" or "default" operands based on the case of an -enum value. This is equivalent to a trivial `switch_enum`_ branch sequence:: - - entry: - switch_enum %0 : $U, \ - case #U.Case1: bb1, \ - case #U.Case2: bb2, /* ... */ \ - default bb_default - bb1: - br cont(%1 : $T) // value for #U.Case1 - bb2: - br cont(%2 : $T) // value for #U.Case2 - bb_default: - br cont(%3 : $T) // value for default - cont(%n : $T): - // use argument %n - -but turns the control flow dependency into a data flow dependency. -For address-only enums, `select_enum_addr`_ offers the same functionality for -an indirectly referenced enum value in memory. - -select_enum_addr -```````````````` -:: - - sil-instruction ::= 'select_enum_addr' sil-operand sil-select-case* - (',' 'default' sil-value)? - ':' sil-type - - %n = select_enum_addr %0 : $*U, \ - case #U.Case1: %1, \ - case #U.Case2: %2, /* ... */ \ - default %3 : $T - - // %0 must be the address of an enum type $*U - // #U.Case1, Case2, etc. must be cases of enum $U - // %1, %2, %3, etc. must have type $T - // %n has type $T - -Selects one of the "case" or "default" operands based on the case of the -referenced enum value. This is the address-only counterpart to -`select_enum`_. - -Protocol and Protocol Composition Types -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -These instructions create and manipulate values of protocol and protocol -composition type. From SIL's perspective, protocol and protocol composition -types consist of an *existential container*, which is a generic container for -a value of unknown runtime type, referred to as an "existential type" in type -theory. The existential container consists of a reference to the -*witness table(s)* for the protocol(s) referred to by the protocol type and a -reference to the underlying *concrete value*, which may be either stored -in-line inside the existential container for small values or allocated -separately into a buffer owned and managed by the existential container for -larger values. - -Depending on the constraints applied to an existential type, an existential -container may use one of several representations: - -- **Opaque existential containers**: If none of the protocols in a protocol - type are class protocols, then the existential container for that type is - address-only and referred to in the implementation as an *opaque existential - container*. The value semantics of the existential container propagate to the - contained concrete value. Applying ``copy_addr`` to an opaque existential - container copies the contained concrete value, deallocating or reallocating - the destination container's owned buffer if necessary. Applying - ``destroy_addr`` to an opaque existential container destroys the concrete - value and deallocates any buffers owned by the existential container. The - following instructions manipulate opaque existential containers: - - * `init_existential_addr`_ - * `open_existential_addr`_ - * `deinit_existential_addr`_ - -- **Class existential containers**: If a protocol type is constrained by one or - more class protocols, then the existential container for that type is - loadable and referred to in the implementation as a *class existential - container*. Class existential containers have reference semantics and can be - ``retain``-ed and ``release``-d. The following instructions manipulate class - existential containers: - - * `init_existential_ref`_ - * `open_existential_ref`_ - -- **Metatype existential containers**: Existential metatypes use a - container consisting of the type metadata for the conforming type along with - the protocol conformances. Metatype existential containers are trivial types. - The following instructions manipulate metatype existential containers: - - * `init_existential_metatype`_ - * `open_existential_metatype`_ - -- **Boxed existential containers**: The standard library ``ErrorType`` protocol - uses a size-optimized reference-counted container, which indirectly stores - the conforming value. Boxed existential containers can be ``retain``-ed - and ``release``-d. The following instructions manipulate boxed existential - containers: - - * `alloc_existential_box`_ - * `open_existential_box`_ - * `dealloc_existential_box`_ - -Some existential types may additionally support specialized representations -when they contain certain known concrete types. For example, when Objective-C -interop is available, the ``ErrorType`` protocol existential supports -a class existential container representation for ``NSError`` objects, so it -can be initialized from one using ``init_existential_ref`` instead of the -more expensive ``alloc_existential_box``:: - - bb(%nserror: $NSError): - // The slow general way to form an ErrorType, allocating a box and - // storing to its value buffer: - %error1 = alloc_existential_box $ErrorType, $NSError - strong_retain %nserror: $NSError - store %nserror to %error1#1 : $NSError - - // The fast path supported for NSError: - strong_retain %nserror: $NSError - %error2 = init_existential_ref %nserror: $NSError, $ErrorType - -init_existential_addr -````````````````````` -:: - - sil-instruction ::= 'init_existential_addr' sil-operand ',' sil-type - - %1 = init_existential_addr %0 : $*P, $T - // %0 must be of a $*P address type for non-class protocol or protocol - // composition type P - // $T must be an AST type that fulfills protocol(s) P - // %1 will be of type $*T', where T' is the maximally abstract lowering - // of type T - -Partially initializes the memory referenced by ``%0`` with an existential -container prepared to contain a value of type ``$T``. The result of the -instruction is an address referencing the storage for the contained value, which -remains uninitialized. The contained value must be ``store``-d or -``copy_addr``-ed to in order for the existential value to be fully initialized. -If the existential container needs to be destroyed while the contained value -is uninitialized, ``deinit_existential_addr`` must be used to do so. A fully -initialized existential container can be destroyed with ``destroy_addr`` as -usual. It is undefined behavior to ``destroy_addr`` a partially-initialized -existential container. - -deinit_existential_addr -``````````````````````` -:: - - sil-instruction ::= 'deinit_existential_addr' sil-operand - - deinit_existential_addr %0 : $*P - // %0 must be of a $*P address type for non-class protocol or protocol - // composition type P - -Undoes the partial initialization performed by -``init_existential_addr``. ``deinit_existential_addr`` is only valid for -existential containers that have been partially initialized by -``init_existential_addr`` but haven't had their contained value initialized. -A fully initialized existential must be destroyed with ``destroy_addr``. - -open_existential_addr -````````````````````` -:: - - sil-instruction ::= 'open_existential_addr' sil-operand 'to' sil-type - - %1 = open_existential_addr %0 : $*P to $*@opened P - // %0 must be of a $*P type for non-class protocol or protocol composition - // type P - // $*@opened P must be a unique archetype that refers to an opened - // existential type P. - // %1 will be of type $*P - -Obtains the address of the concrete value inside the existential -container referenced by ``%0``. The protocol conformances associated -with this existential container are associated directly with the -archetype ``$*@opened P``. This pointer can be used with any operation -on archetypes, such as ``witness_method``. - -init_existential_ref -```````````````````` -:: - - sil-instruction ::= 'init_existential_ref' sil-operand ':' sil-type ',' - sil-type - - %1 = init_existential_ref %0 : $C' : $C, $P - // %0 must be of class type $C', lowered from AST type $C, conforming to - // protocol(s) $P - // $P must be a class protocol or protocol composition type - // %1 will be of type $P - -Creates a class existential container of type ``$P`` containing a reference to -the class instance ``%0``. - -open_existential_ref -```````````````````` -:: - - sil-instruction ::= 'open_existential_ref' sil-operand 'to' sil-type - - %1 = open_existential_ref %0 : $P to $@opened P - // %0 must be of a $P type for a class protocol or protocol composition - // $@opened P must be a unique archetype that refers to an opened - // existential type P - // %1 will be of type $@opened P - -Extracts the class instance reference from a class existential -container. The protocol conformances associated with this existential -container are associated directly with the archetype ``@opened P``. This -pointer can be used with any operation on archetypes, such as -``witness_method``. When the operand is of metatype type, the result -will be the metatype of the opened archetype. - -init_existential_metatype -````````````````````````` -:: - - sil-instruction ::= 'init_existential_metatype' sil-operand ',' sil-type - - %1 = init_existential_metatype $0 : $@ T.Type, $@ P.Type - // %0 must be of a metatype type $@ T.Type where T: P - // %@ P.Type must be the existential metatype of a protocol or protocol - // composition, with the same metatype representation - // %1 will be of type $@ P.Type - -Creates a metatype existential container of type ``$P.Type`` containing the -conforming metatype of ``$T``. - -open_existential_metatype -````````````````````````` -:: - - sil-instruction ::= 'open_existential_metatype' sil-operand 'to' sil-type - - %1 = open_existential_metatype %0 : $@ P.Type to $@ (@opened P).Type - // %0 must be of a $P.Type existential metatype for a protocol or protocol - // composition - // $@ (@opened P).Type must be the metatype of a unique archetype that - // refers to an opened existential type P, with the same metatype - // representation - // %1 will be of type $@ (@opened P).Type - -Extracts the metatype from an existential metatype. The protocol conformances associated with this existential -container are associated directly with the archetype ``@opened P``. - -alloc_existential_box -````````````````````` -:: - - sil-instruction ::= 'alloc_existential_box' sil-type ',' sil-type - - %1 = alloc_existential_box $P, $T - // $P must be a protocol or protocol composition type with boxed - // representation - // $T must be an AST type that conforms to P - // %1#0 will be of type $P - // %1#1 will be of type $*T', where T' is the most abstracted lowering of T - -Allocates a boxed existential container of type ``$P`` with space to hold a -value of type ``$T'``. The box is not fully initialized until a valid value -has been stored into the box. If the box must be deallocated before it is -fully initialized, ``dealloc_existential_box`` must be used. A fully -initialized box can be ``retain``-ed and ``release``-d like any -reference-counted type. The address ``%0#1`` is dependent on the lifetime of -the owner reference ``%0#0``. - -open_existential_box -```````````````````` -:: - - sil-instruction ::= 'open_existential_box' sil-operand 'to' sil-type - - %1 = open_existential_box %0 : $P to $*@opened P - // %0 must be a value of boxed protocol or protocol composition type $P - // %@opened P must be the address type of a unique archetype that refers to - /// an opened existential type P - // %1 will be of type $*@opened P - -Projects the address of the value inside a boxed existential container, and -uses the enclosed type and protocol conformance metadata to bind the -opened archetype ``$@opened P``. The result address is dependent on both -the owning box and the enclosing function; in order to "open" a boxed -existential that has directly adopted a class reference, temporary scratch -space may need to have been allocated. - -dealloc_existential_box -``````````````````````` -:: - - sil-instruction ::= 'dealloc_existential_box' sil-operand, sil-type - - dealloc_existential_box %0 : $P, $T - // %0 must be an uninitialized box of boxed existential container type $P - // $T must be the AST type for which the box was allocated - -Deallocates a boxed existential container. The value inside the existential -buffer is not destroyed; either the box must be uninitialized, or the value -must have been projected out and destroyed beforehand. It is undefined behavior -if the concrete type ``$T`` is not the same type for which the box was -allocated with ``alloc_existential_box``. - -Blocks -~~~~~~ - -project_block_storage -````````````````````` -:: - - sil-instruction ::= 'project_block_storage' sil-operand ':' sil-type - -init_block_storage_header -````````````````````````` - -*TODO* Fill this in. The printing of this instruction looks incomplete on trunk currently. - - -Unchecked Conversions -~~~~~~~~~~~~~~~~~~~~~ - -These instructions implement type conversions which are not checked. These are -either user-level conversions that are always safe and do not need to be -checked, or implementation detail conversions that are unchecked for -performance or flexibility. - -upcast -`````` -:: - - sil-instruction ::= 'upcast' sil-operand 'to' sil-type - - %1 = upcast %0 : $D to $B - // $D and $B must be class types or metatypes, with B a superclass of D - // %1 will have type $B - -Represents a conversion from a derived class instance or metatype to a -superclass, or from a base-class-constrained archetype to its base class. - -address_to_pointer -`````````````````` -:: - - sil-instruction ::= 'address_to_pointer' sil-operand 'to' sil-type - - %1 = address_to_pointer %0 : $*T to $Builtin.RawPointer - // %0 must be of an address type $*T - // %1 will be of type Builtin.RawPointer - -Creates a ``Builtin.RawPointer`` value corresponding to the address ``%0``. -Converting the result pointer back to an address of the same type will give -an address equivalent to ``%0``. It is undefined behavior to cast the -``RawPointer`` to any address type other than its original address type or -any `layout compatible types`_. - -pointer_to_address -`````````````````` -:: - - sil-instruction ::= 'pointer_to_address' sil-operand 'to' sil-type - - %1 = pointer_to_address %0 : $Builtin.RawPointer to $*T - // %1 will be of type $*T - -Creates an address value corresponding to the ``Builtin.RawPointer`` value -``%0``. Converting a ``RawPointer`` back to an address of the same type as -its originating ``address_to_pointer`` instruction gives back an equivalent -address. It is undefined behavior to cast the ``RawPointer`` back to any type -other than its original address type or `layout compatible types`_. It is -also undefined behavior to cast a ``RawPointer`` from a heap object to any -address type. - -unchecked_ref_cast -`````````````````` -:: - - sil-instruction ::= 'unchecked_ref_cast' sil-operand 'to' sil-type - - %1 = unchecked_ref_cast %0 : $A to $B - // %0 must be an object of type $A - // $A must be a type with retainable pointer representation - // %1 will be of type $B - // $B must be a type with retainable pointer representation - -Converts a heap object reference to another heap object reference -type. This conversion is unchecked, and it is undefined behavior if -the destination type is not a valid type for the heap object. The heap -object reference on either side of the cast may be a class -existential, and may be wrapped in one level of Optional. - -unchecked_ref_cast_addr -``````````````````````` -:: - - sil-instruction ::= 'unchecked_ref_cast_addr' - sil-type 'in' sil-operand 'to' - sil-type 'in' sil-operand - - unchecked_ref_cast_addr $A in %0 : $*A to $B in %1 : $*B - // %0 must be the address of an object of type $A - // $A must be a type with retainable pointer representation - // %1 must be the address of storage for an object of type $B - // $B must be a retainable pointer representation - -Loads a heap object reference from an address and stores it at the -address of another uninitialized heap object reference. The loaded -reference is always taken, and the stored reference is -initialized. This conversion is unchecked, and it is undefined -behavior if the destination type is not a valid type for the heap -object. The heap object reference on either side of the cast may be a -class existential, and may be wrapped in one level of Optional. - -unchecked_addr_cast -``````````````````` -:: - - sil-instruction ::= 'unchecked_addr_cast' sil-operand 'to' sil-type - - %1 = unchecked_addr_cast %0 : $*A to $*B - // %0 must be an address - // %1 will be of type $*B - -Converts an address to a different address type. Using the resulting -address is undefined unless ``B`` is layout compatible with ``A``. The -layout of ``A`` may be smaller than that of ``B`` as long as the lower -order bytes have identical layout. - -unchecked_trivial_bit_cast -`````````````````````````` - -:: - - sil-instruction ::= 'unchecked_trivial_bit_cast' sil-operand 'to' sil-type - - %1 = unchecked_trivial_bit_cast %0 : $Builtin.NativeObject to $Builtin.Word - // %0 must be an object. - // %1 must be an object with trivial type. - -Bitcasts an object of type ``A`` to be of same sized or smaller type -``B`` with the constraint that ``B`` must be trivial. This can be used -for bitcasting among trivial types, but more importantly is a one way -bitcast from non-trivial types to trivial types. - -unchecked_bitwise_cast -`````````````````````` -:: - - sil-instruction ::= 'unchecked_bitwise_cast' sil-operand 'to' sil-type - - %1 = unchecked_bitwise_cast %0 : $A to $B - -Bitwise copies an object of type ``A`` into a new object of type ``B`` -of the same size or smaller. - -ref_to_raw_pointer -`````````````````` -:: - - sil-instruction ::= 'ref_to_raw_pointer' sil-operand 'to' sil-type - - %1 = ref_to_raw_pointer %0 : $C to $Builtin.RawPointer - // $C must be a class type, or Builtin.ObjectPointer, or Builtin.ObjCPointer - // %1 will be of type $Builtin.RawPointer - -Converts a heap object reference to a ``Builtin.RawPointer``. The ``RawPointer`` -result can be cast back to the originating class type but does not have -ownership semantics. It is undefined behavior to cast a ``RawPointer`` from a -heap object reference to an address using ``pointer_to_address``. - -raw_pointer_to_ref -`````````````````` -:: - - sil-instruction ::= 'raw_pointer_to_ref' sil-operand 'to' sil-type - - %1 = raw_pointer_to_ref %0 : $Builtin.RawPointer to $C - // $C must be a class type, or Builtin.ObjectPointer, or Builtin.ObjCPointer - // %1 will be of type $C - -Converts a ``Builtin.RawPointer`` back to a heap object reference. Casting -a heap object reference to ``Builtin.RawPointer`` back to the same type gives -an equivalent heap object reference (though the raw pointer has no ownership -semantics for the object on its own). It is undefined behavior to cast a -``RawPointer`` to a type unrelated to the dynamic type of the heap object. -It is also undefined behavior to cast a ``RawPointer`` from an address to any -heap object type. - -ref_to_unowned -`````````````` - -:: - - sil-instruction ::= 'ref_to_unowned' sil-operand - - %1 = unowned_to_ref %0 : T - // $T must be a reference type - // %1 will have type $@unowned T - -Adds the ``@unowned`` qualifier to the type of a reference to a heap -object. No runtime effect. - -unowned_to_ref -`````````````` - -:: - - sil-instruction ::= 'unowned_to_ref' sil-operand - - %1 = unowned_to_ref %0 : $@unowned T - // $T must be a reference type - // %1 will have type $T - -Strips the ``@unowned`` qualifier off the type of a reference to a -heap object. No runtime effect. - -ref_to_unmanaged -```````````````` - -TODO - -unmanaged_to_ref -```````````````` - -TODO - - -convert_function -```````````````` -:: - - sil-instruction ::= 'convert_function' sil-operand 'to' sil-type - - %1 = convert_function %0 : $T -> U to $T' -> U' - // %0 must be of a function type $T -> U ABI-compatible with $T' -> U' - // (see below) - // %1 will be of type $T' -> U' - -Performs a conversion of the function ``%0`` to type ``T``, which must be ABI- -compatible with the type of ``%0``. Function types are ABI-compatible if their -input and result types are tuple types that, after destructuring, differ only -in the following ways: - -- Corresponding tuple elements may add, remove, or change keyword names. - ``(a:Int, b:Float, UnicodeScalar) -> ()`` and ``(x:Int, Float, z:UnicodeScalar) -> ()`` are - ABI compatible. - -- A class tuple element of the destination type may be a superclass or - subclass of the source type's corresponding tuple element. - -The function types may also differ in attributes, with the following -caveats: - -- The ``convention`` attribute cannot be changed. -- A ``@noreturn`` function may be converted to a non-``@noreturn`` - type and vice-versa. - -thin_function_to_pointer -```````````````````````` - -TODO - -pointer_to_thin_function -```````````````````````` - -TODO - -ref_to_bridge_object -```````````````````` -:: - - sil-instruction ::= 'ref_to_bridge_object' sil-operand, sil-operand - - %2 = ref_to_bridge_object %0 : $C, %1 : $Builtin.Word - // %1 must be of reference type $C - // %2 will be of type Builtin.BridgeObject - -Creates a ``Builtin.BridgeObject`` that references ``%0``, with spare bits -in the pointer representation populated by bitwise-OR-ing in the value of -``%1``. It is undefined behavior if this bitwise OR operation affects the -reference identity of ``%0``; in other words, after the following instruction -sequence:: - - %b = ref_to_bridge_object %r : $C, %w : $Builtin.Word - %r2 = bridge_object_to_ref %b : $Builtin.BridgeObject to $C - -``%r`` and ``%r2`` must be equivalent. In particular, it is assumed that -retaining or releasing the ``BridgeObject`` is equivalent to retaining or -releasing the original reference, and that the above ``ref_to_bridge_object`` -/ ``bridge_object_to_ref`` round-trip can be folded away to a no-op. - -On platforms with ObjC interop, there is additionally a platform-specific -bit in the pointer representation of a ``BridgeObject`` that is reserved to -indicate whether the referenced object has native Swift refcounting. It is -undefined behavior to set this bit when the first operand references an -Objective-C object. - -bridge_object_to_ref -```````````````````` -:: - - sil-instruction ::= 'bridge_object_to_ref' sil-operand 'to' sil-type - - %1 = bridge_object_to_ref %0 : $Builtin.BridgeObject to $C - // $C must be a reference type - // %1 will be of type $C - -Extracts the object reference from a ``Builtin.BridgeObject``, masking out any -spare bits. - -bridge_object_to_word -````````````````````` -:: - - sil-instruction ::= 'bridge_object_to_word' sil-operand 'to' sil-type - - %1 = bridge_object_to_word %0 : $Builtin.BridgeObject to $Builtin.Word - // %1 will be of type $Builtin.Word - -Provides the bit pattern of a ``Builtin.BridgeObject`` as an integer. - -thin_to_thick_function -`````````````````````` -:: - - sil-instruction ::= 'thin_to_thick_function' sil-operand 'to' sil-type - - %1 = thin_to_thick_function %0 : $@convention(thin) T -> U to $T -> U - // %0 must be of a thin function type $@convention(thin) T -> U - // The destination type must be the corresponding thick function type - // %1 will be of type $T -> U - -Converts a thin function value, that is, a bare function pointer with no -context information, into a thick function value with ignored context. -Applying the resulting thick function value is equivalent to applying the -original thin value. The ``thin_to_thick_function`` conversion may be -eliminated if the context is proven not to be needed. - -thick_to_objc_metatype -`````````````````````` -:: - - sil-instruction ::= 'thick_to_objc_metatype' sil-operand 'to' sil-type - - %1 = thick_to_objc_metatype %0 : $@thick T.metatype to $@objc_metatype T.metatype - // %0 must be of a thick metatype type $@thick T.metatype - // The destination type must be the corresponding Objective-C metatype type - // %1 will be of type $@objc_metatype T.metatype - -Converts a thick metatype to an Objective-C class metatype. ``T`` must -be of class, class protocol, or class protocol composition type. - -objc_to_thick_metatype -`````````````````````` -:: - - sil-instruction ::= 'objc_to_thick_metatype' sil-operand 'to' sil-type - - %1 = objc_to_thick_metatype %0 : $@objc_metatype T.metatype to $@thick T.metatype - // %0 must be of an Objective-C metatype type $@objc_metatype T.metatype - // The destination type must be the corresponding thick metatype type - // %1 will be of type $@thick T.metatype - -Converts an Objective-C class metatype to a thick metatype. ``T`` must -be of class, class protocol, or class protocol composition type. - -objc_metatype_to_object -``````````````````````` - -TODO - -objc_existential_metatype_to_object -``````````````````````````````````` - -TODO - -is_nonnull -`````````` -:: - - sil-instruction ::= 'is_nonnull' sil-operand - - %1 = is_nonnull %0 : $C - // %0 must be of reference or function type $C - // %1 will be of type Builtin.Int1 - -Checks whether a reference type value is null, returning 1 if -the value is not null, or 0 if it is null. If the value is a function -type, it checks the function pointer (not the data pointer) for null. - -This is not a sensical thing for SIL to represent given that reference -types are non-nullable, but makes sense at the machine level. This is -a horrible hack that should go away someday. - -Checked Conversions -~~~~~~~~~~~~~~~~~~~ - -Some user-level cast operations can fail and thus require runtime checking. - -The `unconditional_checked_cast_addr`_ and `unconditional_checked_cast`_ -instructions performs an unconditional checked cast; it is a runtime failure -if the cast fails. The `checked_cast_addr_br`_ and `checked_cast_br`_ -terminator instruction performs a conditional checked cast; it branches to one -of two destinations based on whether the cast succeeds or not. - -unconditional_checked_cast -`````````````````````````` -:: - - sil-instruction ::= 'unconditional_checked_cast' sil-operand 'to' sil-type - - %1 = unconditional_checked_cast %0 : $A to $B - %1 = unconditional_checked_cast %0 : $*A to $*B - // $A and $B must be both objects or both addresses - // %1 will be of type $B or $*B - -Performs a checked scalar conversion, causing a runtime failure if the -conversion fails. - -unconditional_checked_cast_addr -``````````````````````````````` -:: - - sil-instruction ::= 'unconditional_checked_cast_addr' - sil-cast-consumption-kind - sil-type 'in' sil-operand 'to' - sil-type 'in' sil-operand - sil-cast-consumption-kind ::= 'take_always' - sil-cast-consumption-kind ::= 'take_on_success' - sil-cast-consumption-kind ::= 'copy_on_success' - - %1 = unconditional_checked_cast_addr take_on_success $A in %0 : $*@thick A to $B in $*@thick B - // $A and $B must be both addresses - // %1 will be of type $*B - -Performs a checked indirect conversion, causing a runtime failure if the -conversion fails. - -Runtime Failures -~~~~~~~~~~~~~~~~ - -cond_fail -````````` -:: - - sil-instruction ::= 'cond_fail' sil-operand - - cond_fail %0 : $Builtin.Int1 - // %0 must be of type $Builtin.Int1 - -This instruction produces a `runtime failure`_ if the operand is one. -Execution proceeds normally if the operand is zero. - -Terminators -~~~~~~~~~~~ - -These instructions terminate a basic block. Every basic block must end -with a terminator. Terminators may only appear as the final instruction of -a basic block. - -unreachable -``````````` -:: - - sil-terminator ::= 'unreachable' - - unreachable - -Indicates that control flow must not reach the end of the current basic block. -It is a dataflow error if an unreachable terminator is reachable from the entry -point of a function and is not immediately preceded by an ``apply`` of a -``@noreturn`` function. - -return -`````` -:: - - sil-terminator ::= 'return' sil-operand - - return %0 : $T - // $T must be the return type of the current function - -Exits the current function and returns control to the calling function. If -the current function was invoked with an ``apply`` instruction, the result -of that function will be the operand of this ``return`` instruction. If -the current function was invoked with a ``try_apply` instruction, control -resumes at the normal destination, and the value of the basic block argument -will be the operand of this ``return`` instruction. - -``return`` does not retain or release its operand or any other values. - -A function must not contain more than one ``return`` instruction. - -autorelease_return -`````````````````` -:: - - sil-terminator ::= 'autorelease_return' sil-operand - - autorelease_return %0 : $T - // $T must be the return type of the current function, which must be of - // class type - -Exits the current function and returns control to the calling function. The -result of the ``apply`` instruction that invoked the current function will be -the operand of this ``return`` instruction. The return value is autoreleased -into the active Objective-C autorelease pool using the "autoreleased return -value" optimization. The current function must use the ``@cc(objc_method)`` -calling convention. - -throw -````` -:: - - sil-terminator ::= 'throw' sil-operand - - throw %0 : $T - // $T must be the error result type of the current function - -Exits the current function and returns control to the calling -function. The current function must have an error result, and so the -function must have been invoked with a ``try_apply` instruction. -Control will resume in the error destination of that instruction, and -the basic block argument will be the operand of the ``throw``. - -``throw`` does not retain or release its operand or any other values. - -A function must not contain more than one ``throw`` instruction. - -br -`` -:: - - sil-terminator ::= 'br' sil-identifier - '(' (sil-operand (',' sil-operand)*)? ')' - - br label (%0 : $A, %1 : $B, ...) - // `label` must refer to a basic block label within the current function - // %0, %1, etc. must be of the types of `label`'s arguments - -Unconditionally transfers control from the current basic block to the block -labeled ``label``, binding the given values to the arguments of the destination -basic block. - -cond_br -`````````` -:: - - sil-terminator ::= 'cond_br' sil-operand ',' - sil-identifier '(' (sil-operand (',' sil-operand)*)? ')' ',' - sil-identifier '(' (sil-operand (',' sil-operand)*)? ')' - - cond_br %0 : $Builtin.Int1, true_label (%a : $A, %b : $B, ...), \ - false_label (%x : $X, %y : $Y, ...) - // %0 must be of $Builtin.Int1 type - // `true_label` and `false_label` must refer to block labels within the - // current function and must not be identical - // %a, %b, etc. must be of the types of `true_label`'s arguments - // %x, %y, etc. must be of the types of `false_label`'s arguments - -Conditionally branches to ``true_label`` if ``%0`` is equal to ``1`` or to -``false_label`` if ``%0`` is equal to ``0``, binding the corresponding set of -values to the arguments of the chosen destination block. - -switch_value -```````````` -:: - - sil-terminator ::= 'switch_value' sil-operand - (',' sil-switch-value-case)* - (',' sil-switch-default)? - sil-switch-value-case ::= 'case' sil-value ':' sil-identifier - sil-switch-default ::= 'default' sil-identifier - - switch_value %0 : $Builtin.Int, case %1: label1, \ - case %2: label2, \ - ..., \ - default labelN - - // %0 must be a value of builtin integer type $Builtin.Int - // `label1` through `labelN` must refer to block labels within the current - // function - // FIXME: All destination labels currently must take no arguments - -Conditionally branches to one of several destination basic blocks based on a -value of builtin integer or function type. If the operand value matches one of the ``case`` -values of the instruction, control is transferred to the corresponding basic -block. If there is a ``default`` basic block, control is transferred to it if -the value does not match any of the ``case`` values. It is undefined behavior -if the value does not match any cases and no ``default`` branch is provided. - -select_value -```````````` -:: - - sil-instruction ::= 'select_value' sil-operand sil-select-value-case* - (',' 'default' sil-value)? - ':' sil-type - sil-selct-value-case ::= 'case' sil-value ':' sil-value - - - %n = select_value %0 : $U, \ - case %c1: %r1, \ - case %c2: %r2, /* ... */ \ - default %r3 : $T - - // $U must be a builtin type. Only integers types are supported currently. - // c1, c2, etc must be of type $U - // %r1, %r2, %r3, etc. must have type $T - // %n has type $T - -Selects one of the "case" or "default" operands based on the case of an -value. This is equivalent to a trivial `switch_value`_ branch sequence:: - - entry: - switch_value %0 : $U, \ - case %c1: bb1, \ - case %c2: bb2, /* ... */ \ - default bb_default - bb1: - br cont(%r1 : $T) // value for %c1 - bb2: - br cont(%r2 : $T) // value for %c2 - bb_default: - br cont(%r3 : $T) // value for default - cont(%n : $T): - // use argument %n - -but turns the control flow dependency into a data flow dependency. - -switch_enum -``````````` -:: - - sil-terminator ::= 'switch_enum' sil-operand - (',' sil-switch-enum-case)* - (',' sil-switch-default)? - sil-switch-enum-case ::= 'case' sil-decl-ref ':' sil-identifier - - switch_enum %0 : $U, case #U.Foo: label1, \ - case #U.Bar: label2, \ - ..., \ - default labelN - - // %0 must be a value of enum type $U - // #U.Foo, #U.Bar, etc. must be 'case' declarations inside $U - // `label1` through `labelN` must refer to block labels within the current - // function - // label1 must take either no basic block arguments, or a single argument - // of the type of #U.Foo's data - // label2 must take either no basic block arguments, or a single argument - // of the type of #U.Bar's data, etc. - // labelN must take no basic block arguments - -Conditionally branches to one of several destination basic blocks based on the -discriminator in a loadable ``enum`` value. Unlike ``switch_int``, -``switch_enum`` requires coverage of the operand type: If the ``enum`` type -is resilient, the ``default`` branch is required; if the ``enum`` type is -fragile, the ``default`` branch is required unless a destination is assigned to -every ``case`` of the ``enum``. The destination basic block for a ``case`` may -take an argument of the corresponding ``enum`` ``case``'s data type (or of the -address type, if the operand is an address). If the branch is taken, the -destination's argument will be bound to the associated data inside the -original enum value. For example:: - - enum Foo { - case Nothing - case OneInt(Int) - case TwoInts(Int, Int) - } - - sil @sum_of_foo : $Foo -> Int { - entry(%x : $Foo): - switch_enum %x : $Foo, \ - case #Foo.Nothing: nothing, \ - case #Foo.OneInt: one_int, \ - case #Foo.TwoInts: two_ints - - nothing: - %zero = integer_literal 0 : $Int - return %zero : $Int - - one_int(%y : $Int): - return %y : $Int - - two_ints(%ab : $(Int, Int)): - %a = tuple_extract %ab : $(Int, Int), 0 - %b = tuple_extract %ab : $(Int, Int), 1 - %add = function_ref @add : $(Int, Int) -> Int - %result = apply %add(%a, %b) : $(Int, Int) -> Int - return %result : $Int - } - -On a path dominated by a destination block of ``switch_enum``, copying or -destroying the basic block argument has equivalent reference counting semantics -to copying or destroying the ``switch_enum`` operand:: - - // This retain_value... - retain_value %e1 : $Enum - switch_enum %e1, case #Enum.A: a, case #Enum.B: b - a(%a : $A): - // ...is balanced by this release_value - release_value %a - b(%b : $B): - // ...and this one - release_value %b - -switch_enum_addr -```````````````` -:: - - sil-terminator ::= 'switch_enum_addr' sil-operand - (',' sil-switch-enum-case)* - (',' sil-switch-default)? - - switch_enum_addr %0 : $*U, case #U.Foo: label1, \ - case #U.Bar: label2, \ - ..., \ - default labelN - - // %0 must be the address of an enum type $*U - // #U.Foo, #U.Bar, etc. must be cases of $U - // `label1` through `labelN` must refer to block labels within the current - // function - // The destinations must take no basic block arguments - -Conditionally branches to one of several destination basic blocks based on -the discriminator in the enum value referenced by the address operand. - -Unlike ``switch_int``, ``switch_enum`` requires coverage of the operand type: -If the ``enum`` type is resilient, the ``default`` branch is required; if the -``enum`` type is fragile, the ``default`` branch is required unless a -destination is assigned to every ``case`` of the ``enum``. -Unlike ``switch_enum``, the payload value is not passed to the destination -basic blocks; it must be projected out separately with `unchecked_take_enum_data_addr`_. - -dynamic_method_br -````````````````` -:: - - sil-terminator ::= 'dynamic_method_br' sil-operand ',' sil-decl-ref - ',' sil-identifier ',' sil-identifier - - dynamic_method_br %0 : $P, #X.method!1, bb1, bb2 - // %0 must be of protocol type - // #X.method!1 must be a reference to an @objc method of any class - // or protocol type - -Looks up the implementation of an Objective-C method with the same -selector as the named method for the dynamic type of the value inside -an existential container. The "self" operand of the result function -value is represented using an opaque type, the value for which must be -projected out as a value of type ``Builtin.ObjCPointer``. - -If the operand is determined to have the named method, this -instruction branches to ``bb1``, passing it the uncurried function -corresponding to the method found. If the operand does not have the -named method, this instruction branches to ``bb2``. - -checked_cast_br -``````````````` -:: - - sil-terminator ::= 'checked_cast_br' sil-checked-cast-exact? - sil-operand 'to' sil-type ',' - sil-identifier ',' sil-identifier - sil-checked-cast-exact ::= '[' 'exact' ']' - - checked_cast_br %0 : $A to $B, bb1, bb2 - checked_cast_br %0 : $*A to $*B, bb1, bb2 - checked_cast_br [exact] %0 : $A to $A, bb1, bb2 - // $A and $B must be both object types or both address types - // bb1 must take a single argument of type $B or $*B - // bb2 must take no arguments - -Performs a checked scalar conversion from ``$A`` to ``$B``. If the conversion -succeeds, control is transferred to ``bb1``, and the result of the cast is -passed into ``bb1`` as an argument. If the conversion fails, control is -transferred to ``bb2``. - -An exact cast checks whether the dynamic type is exactly the target -type, not any possible subtype of it. The source and target types -must be class types. - -checked_cast_addr_br -```````````````````` -:: - - sil-terminator ::= 'checked_cast_addr_br' - sil-cast-consumption-kind - sil-type 'in' sil-operand 'to' - sil-stype 'in' sil-operand ',' - sil-identifier ',' sil-identifier - sil-cast-consumption-kind ::= 'take_always' - sil-cast-consumption-kind ::= 'take_on_success' - sil-cast-consumption-kind ::= 'copy_on_success' - - checked_cast_addr_br take_always $A in %0 : $*@thick A to $B in %2 : $*@thick B, bb1, bb2 - // $A and $B must be both address types - // bb1 must take a single argument of type $*B - // bb2 must take no arguments - -Performs a checked indirect conversion from ``$A`` to ``$B``. If the -conversion succeeds, control is transferred to ``bb1``, and the result of the -cast is left in the destination. If the conversion fails, control is -transferred to ``bb2``. - -try_apply -````````` -:: - - sil-terminator ::= 'try_apply' sil-value - sil-apply-substitution-list? - '(' (sil-value (',' sil-value)*)? ')' - ':' sil-type - 'normal' sil-identifier, 'error' sil-identifier - - try_apply %0(%1, %2, ...) : $(A, B, ...) -> (R, @error E), - normal bb1, error bb2 - bb1(%3 : R): - bb2(%4 : E): - - // Note that the type of the callee '%0' is specified *after* the arguments - // %0 must be of a concrete function type $(A, B, ...) -> (R, @error E) - // %1, %2, etc. must be of the argument types $A, $B, etc. - -Transfers control to the function specified by ``%0``, passing it the -given arguments. When ``%0`` returns, control resumes in either the -normal destination (if it returns with ``return``) or the error -destination (if it returns with ``throw``). - -``%0`` must have a function type with an error result. - -The rules on generic substitutions are identical to those of ``apply``. - -Assertion configuration -~~~~~~~~~~~~~~~~~~~~~~~ - -To be able to support disabling assertions at compile time there is a builtin -``assertion_configuration`` function. A call to this function can be replaced at -compile time by a constant or can stay opaque. - -All calls to the ``assert_configuration`` function are replaced by the constant -propagation pass to the appropriate constant depending on compile time settings. -Subsequent passes remove dependent unwanted control flow. Using this mechanism -we support conditionally enabling/disabling of code in SIL libraries depending -on the assertion configuration selected when the library is linked into user -code. - -There are three assertion configurations: Debug (0), Release (1) and -DisableReplacement (-1). - -The optimization flag or a special assert configuration flag determines the -value. Depending on the configuration value assertions in the standard library -will be executed or not. - -The standard library uses this builtin to define an assert that can be -disabled at compile time. - -:: - - func assert(...) { - if (Int32(Builtin.assert_configuration()) == 0) { - _fatal_error_message(message, ...) - } - } - -The ``assert_configuration`` function application is serialized when we build -the standard library (we recognize the ``-parse-stdlib`` option and don't do the -constant replacement but leave the function application to be serialized to -sil). - -The compiler flag that influences the value of the ``assert_configuration`` -funtion application is the optimization flag: at ``-Onone` the application will -be replaced by ``Debug`` at higher optimization levels the instruction will be -replaced by ``Release``. Optionally, the value to use for replacement can be -specified with the ``-AssertConf`` flag which overwrites the value selected by -the optimization flag (possible values are ``Debug``, ``Release``, -``DisableReplacement``). - -If the call to the ``assert_configuration`` function stays opaque until IRGen, -IRGen will replace the application by the constant representing Debug mode (0). -This happens we can build the standard library .dylib. The generate sil will -retain the function call but the generated .dylib will contain code with -assertions enabled. diff --git a/docs/SequencesAndCollections.md b/docs/SequencesAndCollections.md new file mode 100644 index 0000000000000..961c455829f18 --- /dev/null +++ b/docs/SequencesAndCollections.md @@ -0,0 +1,372 @@ +Sequences And Collections in Swift +================================== + +Unlike many languages, Swift provides a rich taxonomy of abstractions +for processing series of elements. This document explains why that +taxonomy exists and how it is structured. + +Sequences +--------- + +It all begins with Swift's `for`{.sourceCode}…`in`{.sourceCode} loop: + + for x in s { + doSomethingWith(x) + } + +Because this construct is generic, `s`{.sourceCode} could be + +- an array +- a set +- a linked list +- a series of UI events +- a file on disk +- a stream of incoming network packets +- an infinite series of random numbers +- a user-defined data structure +- etc. + +In Swift, all of the above are called **sequences**, an abstraction +represented by the `SequenceType`{.sourceCode} protocol: + + protocol SequenceType { + typealias Generator : GeneratorType + func generate() -> Generator + } + +> **Hiding Generator Type Details** +> +> A sequence's generator is an associated type—rather than something +> like |AnyGenerator|\_\_ that depends only on the element type—for +> performance reasons. Although the alternative design has significant +> usability benefits, it requires one dynamic allocation/deallocation +> pair and *N* dynamic dispatches to traverse a sequence of length *N*. +> That said, our optimizer has improved to the point where it can +> sometimes remove these overheads completely, and we are +> [considering](rdar://19755076) changing the design accordingly. +> +> \_\_ + +As you can see, sequence does nothing more than deliver a generator. To +understand the need for generators, it's important to distinguish the +two kinds of sequences. + +- **Volatile** sequences like “stream of network packets,” carry their + own traversal state, and are expected to be “consumed” as they + are traversed. +- **Stable** sequences, like arrays, should *not* be mutated by + `for`{.sourceCode} + …`in`{.sourceCode}, and thus require *separate traversal state*. + +To get an initial traversal state for an arbitrary sequence +`x`{.sourceCode}, Swift calls `x.generate()`{.sourceCode}. The sequence +delivers that state, along with traversal logic, in the form of a +**generator**. + +Generators +---------- + +`for`{.sourceCode}…`in`{.sourceCode} needs three operations from the +generator: + +- get the current element +- advance to the next element +- detect whether there are more elements + +If we literally translate the above into protocol requirements, we get +something like this: + + protocol NaiveGeneratorType { + typealias Element + var current() -> Element // get the current element + mutating func advance() // advance to the next element + var isExhausted: Bool // detect whether there are more elements + } + +Such a protocol, though, places a burden on implementors of volatile +sequences: either the generator must buffer the current element +internally so that `current`{.sourceCode} can repeatedly return the same +value, or it must trap when `current`{.sourceCode} is called twice +without an intervening call to `moveToNext`{.sourceCode}. Both semantics +have a performance cost, and the latter unnecessarily adds the +possibility of incorrect usage. + +> **`NSEnumerator`{.sourceCode}** +> +> You might recognize the influence on generators of the +> `NSEnumerator`{.sourceCode} API: +> +> class NSEnumerator : NSObject { +> func nextObject() -> AnyObject? +> } + +Therefore, Swift's `GeneratorType`{.sourceCode} merges the three +operations into one, returning `nil`{.sourceCode} when the generator is +exhausted: + + protocol GeneratorType { + typealias Element + mutating func next() -> Element? + } + +Combined with `SequenceType`{.sourceCode}, we now have everything we +need to implement a generic `for`{.sourceCode}…`in`{.sourceCode} loop. + +> **Adding a Buffer** +> +> The use-cases for singly-buffered generators are rare enough that it +> is not worth complicating `GeneratorType`{.sourceCode}, [^1] but +> support for buffering would fit nicely into the scheme, should it +> prove important: +> +> public protocol BufferedGeneratorType +> : GeneratorType { +> var latest: Element? {get} +> } +> +> The library could easily offer a generic wrapper that adapts any +> `GeneratorType`{.sourceCode} to create a \`BufferedGeneratorType\`: +> +> /// Add buffering to any GeneratorType G +> struct BufferedGenerator +> : BufferedGeneratorType { +> +> public init(_ baseGenerator: G) { +> self._baseGenerator = baseGenerator +> } +> public func next() -> Element? { +> latest = _baseGenerator.next() ?? latest +> return latest +> } +> public private(set) var +> latest: G.Element? = nil +> private var _baseGenerator: G +> } + +### Operating on Sequences Generically + +Given an arbitrary `SequenceType`{.sourceCode}, aside from a simple +`for`{.sourceCode}…`in`{.sourceCode} loop, you can do anything that +requires reading elements from beginning to end. For example: + + // Return an array containing the elements of `source`, with + // `separator` interposed between each consecutive pair. + func array( + source: S, + withSeparator separator: S.Generator.Element + ) -> [S.Generator.Element] { + var result: [S.Generator.Element] = [] + var g = source.generate() + if let start = g.next() { + result.append(start) + while let next = g.next() { + result.append(separator) + result.append(next) + } + } + return result + } + + let s = String(array("Swift", withSeparator: "|")) + print(s) // "S|w|i|f|t" + +Because sequences may be volatile, though, you can—in general—only make +a single traversal. This capability is quite enough for many languages: +the iteration abstractions of Java, C\#, Python, and Ruby all go about +as far as `SequenceType`{.sourceCode}, and no further. In Swift, though, +we want to do much more generically. All of the following depend on +stability that an arbitrary sequence can't provide: + +- Finding a sub-sequence +- Finding the element that occurs most often +- Meaningful in-place element mutation (including sorting, + partitioning, rotations, etc.) + +> **Generators Should Be Sequences** +> +> In principle, every generator is a volatile sequence containing the +> elements it has yet to return from `next()`{.sourceCode}. Therefore, +> every generator *could* satisfy the requirements of +> `SequenceType`{.sourceCode} by simply declaring conformance, and +> returning `self`{.sourceCode} from its `generate()`{.sourceCode} +> method. In fact, if it weren't for [current language +> limitations](rdar://17986597), `GeneratorType`{.sourceCode} would +> refine `SequenceType`{.sourceCode}, as follows: +> +> Though we may not currently be able to *require* that every +> `GeneratorType`{.sourceCode} refines `SequenceType`{.sourceCode}, most +> generators in the standard library do conform to +> `SequenceType`{.sourceCode}. + +Fortunately, many real sequences *are* stable. To take advantage of that +stability in generic code, we'll need another protocol. + +Collections +----------- + +A **collection** is a stable sequence with addressable “positions,” +represented by an associated `Index`{.sourceCode} type: + + protocol CollectionType : SequenceType { + typealias Index : ForwardIndexType // a position + subscript(i: Index) -> Generator.Element {get} + + var startIndex: Index {get} + var endIndex: Index {get} + } + +The way we address positions in a collection is a generalization of how +we interact with arrays: we subscript the collection using its +`Index`{.sourceCode} type: + + let ith = c[i] + +An **index**—which must model `ForwardIndexType`{.sourceCode}—is a type +with a linear series of discrete values that can be compared for +equality: + +> **Dictionary Keys** +> +> Although dictionaries overload `subscript`{.sourceCode} to also +> operate on keys, a `Dictionary`{.sourceCode}'s `Key`{.sourceCode} type +> is distinct from its `Index`{.sourceCode} type. Subscripting on an +> index is expected to offer direct access, without introducing +> overheads like searching or hashing. + + protocol ForwardIndexType : Equatable { + typealias Distance : SignedIntegerType + func successor() -> Self + } + +While one can use `successor()`{.sourceCode} to create an incremented +index value, indices are more commonly advanced using an in-place +increment operator, just as one would when traversing an array: +`++i`{.sourceCode} or `i++`{.sourceCode}. These operators are defined +generically, for all models of `ForwardIndexType`{.sourceCode}, in terms +of the `successor()`{.sourceCode} method. + +Every collection has two special indices: a `startIndex`{.sourceCode} +and an `endIndex`{.sourceCode}. In an empty collection, +`startIndex == endIndex`{.sourceCode}. Otherwise, +`startIndex`{.sourceCode} addresses the collection's first element, and +`endIndex`{.sourceCode} is the successor of an index addressing the +collection's last element. A collection's `startIndex`{.sourceCode} and +`endIndex`{.sourceCode} form a half-open range containing its elements: +while a collection's `endIndex`{.sourceCode} is a valid index value for +comparison, it is not a valid index for subscripting the collection: + + if c.startIndex != c.endIndex { } // OK + c[c.endIndex] // Oops! (index out-of-range) + +### Mutable Collections + +A **mutable collection** is a collection that supports in-place element +mutation. The protocol is a simple refinement of +`CollectionType`{.sourceCode} that adds a subscript setter: + +The `CollectionType`{.sourceCode} protocol does not require collection +to support mutation, so it is not possible to tell from the protocol +itself whether the order of elements in an instance of a type that +conforms to `CollectionType`{.sourceCode} has a domain-specific meaning +or not. (Note that since elements in collections have stable indices, +the element order within the collection itself is stable; the order +sometimes does not have a meaning and is not chosen by the code that +uses the collection, but by the implementation details of the collection +itself.) + +`MutableCollectionType`{.sourceCode} protocol allows the to replace a +specific element, identified by an index, with another one in the same +position. This capability essentially allows to rearrange the elements +inside the collection in any order, thus types that conform to +`MutableCollectionType`{.sourceCode} can represent collections with a +domain-specific element order (not every instance of a +`MutableCollectionType`{.sourceCode} has an interesting order, though). + +### Range Replaceable Collections + +The `MutableCollectionType`{.sourceCode} protocol implies only mutation +of content, not of structure (for example, changing the number of +elements). The `RangeReplaceableCollectionType`{.sourceCode} protocol +adds the capability to perform structural mutation, which in its most +general form is expressed as replacing a range of elements, denoted by +two indices, by elements from a collection with a **different** length. + + public protocol RangeReplaceableCollectionType : MutableCollectionType { + mutating func replaceRange< + C: CollectionType where C.Generator.Element == Self.Generator.Element + >( + subRange: Range, with newElements: C + ) + } + +### Index Protocols + +As a generalization designed to cover diverse data structures, +`CollectionType`{.sourceCode} provides weaker guarantees than arrays do. +In particular, an arbitrary collection does not necessarily offer +efficient random access; that property is determined by the protocol +conformances of its `Index`{.sourceCode} type. + +**Forward indices** are the simplest and most general, capturing the +capabilities of indices into a singly-linked list: + +1. advance to the next position +2. detect the end position + +**Bidirectional indices** are a refinement of forward indices that +additionally support reverse traversal: + + protocol BidirectionalIndexType : ForwardIndexType { + func predecessor() -> Self + } + +Indices into a doubly-linked list would be bidirectional, as are the +indices that address `Character`{.sourceCode}s and +`UnicodeScalar`{.sourceCode}s in a `String`{.sourceCode}. Reversing the +order of a collection's elements is a simple example of a generic +algorithm that depends on bidirectional traversal. + +**Random access indices** have two more requirements: the ability to +efficiently measure the number of steps between arbitrary indices +addressing the same collection, and the ability to advance an index by a +(possibly negative) number of steps: + + public protocol RandomAccessIndexType : BidirectionalIndexType { + func distanceTo(other: Self) -> Distance + func advancedBy(n: Distance) -> Self + } + +From these methods, the standard library derives several other features +such as `Comparable`{.sourceCode} conformance, index subtraction, and +addition/subtraction of integers to/from indices. + +The indices of a +[deque](http://en.wikipedia.org/wiki/Double-ended_queue) can provide +random access, as do the indices into `String.UTF16View`{.sourceCode} +(when Foundation is loaded) and, of course, array indices. Many common +sorting and selection algorithms, among others, depend on these +capabilities. + +All direct operations on indices are intended to be lightweight, with +amortized O(1) complexity. In fact, indices into +`Dictionary`{.sourceCode} and `Set`{.sourceCode} *could* be +bidirectional, but are limited to modeling +`ForwardIndexType`{.sourceCode} because the APIs of +`NSDictionary`{.sourceCode} and `NSSet`{.sourceCode}—which can act as +backing stores of `Dictionary`{.sourceCode} and `Set`{.sourceCode}—do +not efficiently support reverse traversal. + +Conclusion +---------- + +Swift's sequence, collection, and index protocols allow us to write +general algorithms that apply to a wide variety of series and data +structures. The system has been both easy to extend, and predictably +performant. Thanks for taking the tour! + +------------------------------------------------------------------------ + +[^1]: This trade-off is not as obvious as it might seem. For example, + the C\# and C++ analogues for `GeneratorType`{.sourceCode} + (`IEnumerable`{.sourceCode} and `input iterator`{.sourceCode}) are + saddled with the obligation to provide buffering. diff --git a/docs/SequencesAndCollections.rst b/docs/SequencesAndCollections.rst deleted file mode 100644 index f6272314b22e1..0000000000000 --- a/docs/SequencesAndCollections.rst +++ /dev/null @@ -1,386 +0,0 @@ -:orphan: - -.. @raise litre.TestsAreMissing -.. default-role:: code - -==================================== - Sequences And Collections in Swift -==================================== - -Unlike many languages, Swift provides a rich taxonomy of abstractions -for processing series of elements. This document explains why that -taxonomy exists and how it is structured. - -Sequences -========= - -It all begins with Swift's `for`\ …\ `in` loop:: - - for x in s { - doSomethingWith(x) - } - -Because this construct is generic, `s` could be - -* an array -* a set -* a linked list -* a series of UI events -* a file on disk -* a stream of incoming network packets -* an infinite series of random numbers -* a user-defined data structure -* etc. - -In Swift, all of the above are called **sequences**, an abstraction -represented by the `SequenceType` protocol:: - - protocol SequenceType { - typealias Generator : GeneratorType - func generate() -> Generator - } - -.. sidebar:: Hiding Generator Type Details - - A sequence's generator is an associated type—rather than something - like |AnyGenerator|__ that depends only on the element type—for - performance reasons. Although the alternative design has - significant usability benefits, it requires one dynamic - allocation/deallocation pair and *N* dynamic dispatches to traverse - a sequence of length *N*. That said, our optimizer has improved to - the point where it can sometimes remove these overheads completely, - and we are `considering `_ changing the design - accordingly. - - .. |AnyGenerator| replace:: `AnyGenerator` - - __ http://swiftdoc.org/type/AnyGenerator/ - -As you can see, sequence does nothing more than deliver a generator. -To understand the need for generators, it's important to distinguish -the two kinds of sequences. - -* **Volatile** sequences like “stream of network packets,” carry - their own traversal state, and are expected to be “consumed” as they - are traversed. - -* **Stable** sequences, like arrays, should *not* be mutated by `for`\ - …\ `in`, and thus require *separate traversal state*. - -To get an initial traversal state for an arbitrary sequence `x`, Swift -calls `x.generate()`. The sequence delivers that state, along with -traversal logic, in the form of a **generator**. - -Generators -========== - -`for`\ …\ `in` needs three operations from the generator: - -* get the current element -* advance to the next element -* detect whether there are more elements - -If we literally translate the above into protocol requirements, we get -something like this:: - - protocol NaiveGeneratorType { - typealias Element - var current() -> Element // get the current element - mutating func advance() // advance to the next element - var isExhausted: Bool // detect whether there are more elements - } - -Such a protocol, though, places a burden on implementors of volatile -sequences: either the generator must buffer the current element -internally so that `current` can repeatedly return the same value, or -it must trap when `current` is called twice without an intervening -call to `moveToNext`. Both semantics have a performance cost, and -the latter unnecessarily adds the possibility of incorrect usage. - -.. sidebar:: `NSEnumerator` - - You might recognize the influence on generators of the - `NSEnumerator` API:: - - class NSEnumerator : NSObject { - func nextObject() -> AnyObject? - } - -Therefore, Swift's `GeneratorType` merges the three operations into one, -returning `nil` when the generator is exhausted:: - - protocol GeneratorType { - typealias Element - mutating func next() -> Element? - } - -Combined with `SequenceType`, we now have everything we need to -implement a generic `for`\ …\ `in` loop. - -.. sidebar:: Adding a Buffer - - The use-cases for singly-buffered generators are rare enough that it - is not worth complicating `GeneratorType`, [#input_iterator]_ but - support for buffering would fit nicely into the scheme, should it - prove important:: - - public protocol BufferedGeneratorType - : GeneratorType { - var latest: Element? {get} - } - - The library could easily offer a generic wrapper that adapts any - `GeneratorType` to create a `BufferedGeneratorType`:: - - /// Add buffering to any GeneratorType G - struct BufferedGenerator - : BufferedGeneratorType { - - public init(_ baseGenerator: G) { - self._baseGenerator = baseGenerator - } - public func next() -> Element? { - latest = _baseGenerator.next() ?? latest - return latest - } - public private(set) var - latest: G.Element? = nil - private var _baseGenerator: G - } - -Operating on Sequences Generically ----------------------------------- - -Given an arbitrary `SequenceType`, aside from a simple `for`\ …\ `in` loop, -you can do anything that requires reading elements from beginning to -end. For example:: - - // Return an array containing the elements of `source`, with - // `separator` interposed between each consecutive pair. - func array( - source: S, - withSeparator separator: S.Generator.Element - ) -> [S.Generator.Element] { - var result: [S.Generator.Element] = [] - var g = source.generate() - if let start = g.next() { - result.append(start) - while let next = g.next() { - result.append(separator) - result.append(next) - } - } - return result - } - - let s = String(array("Swift", withSeparator: "|")) - print(s) // "S|w|i|f|t" - -Because sequences may be volatile, though, you can—in general—only -make a single traversal. This capability is quite enough for many -languages: the iteration abstractions of Java, C#, Python, and Ruby -all go about as far as `SequenceType`, and no further. In Swift, -though, we want to do much more generically. All of the following -depend on stability that an arbitrary sequence can't provide: - -* Finding a sub-sequence -* Finding the element that occurs most often -* Meaningful in-place element mutation (including sorting, - partitioning, rotations, etc.) - -.. sidebar:: Generators Should Be Sequences - - In principle, every generator is a volatile sequence containing - the elements it has yet to return from `next()`. Therefore, every - generator *could* satisfy the requirements of `SequenceType` by - simply declaring conformance, and returning `self` from its - `generate()` method. In fact, if it weren't for `current language - limitations `_, `GeneratorType` would refine - `SequenceType`, as follows: - - .. parsed-literal:: - - protocol GeneratorType **: SequenceType** { - typealias Element - mutating func next() -> Element? - } - - Though we may not currently be able to *require* that every - `GeneratorType` refines `SequenceType`, most generators in the - standard library do conform to `SequenceType`. - -Fortunately, many real sequences *are* stable. To take advantage of -that stability in generic code, we'll need another protocol. - -Collections -=========== - -A **collection** is a stable sequence with addressable “positions,” -represented by an associated `Index` type:: - - protocol CollectionType : SequenceType { - typealias Index : ForwardIndexType // a position - subscript(i: Index) -> Generator.Element {get} - - var startIndex: Index {get} - var endIndex: Index {get} - } - -The way we address positions in a collection is a generalization of -how we interact with arrays: we subscript the collection using its -`Index` type:: - - let ith = c[i] - -An **index**\ —which must model `ForwardIndexType`\ —is a type with a -linear series of discrete values that can be compared for equality: - -.. sidebar:: Dictionary Keys - - Although dictionaries overload `subscript` to also operate on keys, - a `Dictionary`\ 's `Key` type is distinct from its `Index` type. - Subscripting on an index is expected to offer direct access, - without introducing overheads like searching or hashing. - -:: - - protocol ForwardIndexType : Equatable { - typealias Distance : SignedIntegerType - func successor() -> Self - } - -While one can use `successor()` to create an incremented index value, -indices are more commonly advanced using an in-place increment -operator, just as one would when traversing an array: `++i` or `i++`. -These operators are defined generically, for all models of -`ForwardIndexType`, in terms of the `successor()` method. - -Every collection has two special indices: a `startIndex` and an -`endIndex`. In an empty collection, `startIndex == endIndex`. -Otherwise, `startIndex` addresses the collection's first element, and -`endIndex` is the successor of an index addressing the collection's -last element. A collection's `startIndex` and `endIndex` form a -half-open range containing its elements: while a collection's -`endIndex` is a valid index value for comparison, it is not a valid -index for subscripting the collection:: - - if c.startIndex != c.endIndex { } // OK - c[c.endIndex] // Oops! (index out-of-range) - -Mutable Collections -------------------- - -A **mutable collection** is a collection that supports in-place element -mutation. The protocol is a simple refinement of `CollectionType` that adds a -subscript setter: - -.. parsed-literal:: - - protocol MutableCollectionType : CollectionType { - subscript(i: Index) -> Generator.Element { get **set** } - } - -The `CollectionType` protocol does not require collection to support mutation, -so it is not possible to tell from the protocol itself whether the order of -elements in an instance of a type that conforms to `CollectionType` has a -domain-specific meaning or not. (Note that since elements in collections have -stable indices, the element order within the collection itself is stable; the -order sometimes does not have a meaning and is not chosen by the code that uses -the collection, but by the implementation details of the collection itself.) - -`MutableCollectionType` protocol allows the to replace a specific element, -identified by an index, with another one in the same position. This capability -essentially allows to rearrange the elements inside the collection in any -order, thus types that conform to `MutableCollectionType` can represent -collections with a domain-specific element order (not every instance of a -`MutableCollectionType` has an interesting order, though). - -Range Replaceable Collections ------------------------------ - -The `MutableCollectionType` protocol implies only mutation of content, not of -structure (for example, changing the number of elements). The -`RangeReplaceableCollectionType` protocol adds the capability to perform -structural mutation, which in its most general form is expressed as replacing a -range of elements, denoted by two indices, by elements from a collection with a -**different** length. - -:: - - public protocol RangeReplaceableCollectionType : MutableCollectionType { - mutating func replaceRange< - C: CollectionType where C.Generator.Element == Self.Generator.Element - >( - subRange: Range, with newElements: C - ) - } - - -Index Protocols ---------------- - -As a generalization designed to cover diverse data structures, -`CollectionType` provides weaker guarantees than arrays do. In -particular, an arbitrary collection does not necessarily offer -efficient random access; that property is determined by the protocol -conformances of its `Index` type. - -**Forward indices** are the simplest and most general, capturing the -capabilities of indices into a singly-linked list: - -1. advance to the next position -2. detect the end position - -**Bidirectional indices** are a refinement of forward indices that -additionally support reverse traversal:: - - protocol BidirectionalIndexType : ForwardIndexType { - func predecessor() -> Self - } - -Indices into a doubly-linked list would be bidirectional, as are the -indices that address `Character`\ s and `UnicodeScalar`\ s in a -`String`. Reversing the order of a collection's elements is a simple -example of a generic algorithm that depends on bidirectional traversal. - -**Random access indices** have two more requirements: the ability to -efficiently measure the number of steps between arbitrary indices -addressing the same collection, and the ability to advance an index by -a (possibly negative) number of steps:: - - public protocol RandomAccessIndexType : BidirectionalIndexType { - func distanceTo(other: Self) -> Distance - func advancedBy(n: Distance) -> Self - } - -From these methods, the standard library derives several other -features such as `Comparable` conformance, index subtraction, and -addition/subtraction of integers to/from indices. - -The indices of a `deque -`_ can provide random -access, as do the indices into `String.UTF16View` (when Foundation is -loaded) and, of course, array indices. Many common sorting and -selection algorithms, among others, depend on these capabilities. - -All direct operations on indices are intended to be lightweight, with -amortized O(1) complexity. In fact, indices into `Dictionary` and -`Set` *could* be bidirectional, but are limited to modeling -`ForwardIndexType` because the APIs of `NSDictionary` and -`NSSet`—which can act as backing stores of `Dictionary` and `Set`—do -not efficiently support reverse traversal. - -Conclusion -========== - -Swift's sequence, collection, and index protocols allow us to write -general algorithms that apply to a wide variety of series and data -structures. The system has been both easy to extend, and predictably -performant. Thanks for taking the tour! - ------- - -.. [#input_iterator] This trade-off is not as obvious as it might - seem. For example, the C# and C++ analogues for `GeneratorType` - (`IEnumerable` and `input iterator`) are saddled with the - obligation to provide buffering. diff --git a/docs/Serialization.md b/docs/Serialization.md new file mode 100644 index 0000000000000..d19d40753bf5b --- /dev/null +++ b/docs/Serialization.md @@ -0,0 +1,155 @@ +Swift Binary Serialization Format +================================= + +The fundamental unit of distribution for Swift code is a *module.* A +module contains declarations as an interface for clients to write code +against. It may also contain implementation information for any of these +declarations that can be used to optimize client code. Conceptually, the +file containing the interface for a module serves much the same purpose +as the collection of C header files for a particular library. + +Swift's binary serialization format is currently used for several +purposes: + +- The public interface for a module ("swiftmodule files"). +- A representation of captured compiler state after semantic analysis + and SIL generation, but before LLVM IR generation ("SIB", for "Swift + Intermediate Binary"). +- Debug information about types, for proper high-level introspection + without running code. +- Debug information about non-public APIs, for interactive debugging. + +The first two uses require a module to serve as a container of both AST +nodes and SIL entities. As a unit of distribution, it should also be +forward-compatible: module files installed on a developer's system in +201X should be usable without updates for years to come, even as the +Swift compiler continues to be improved and enhanced. However, they are +currently too closely tied to the compiler internals to be useful for +this purpose, and it is likely we'll invent a new format instead. + +Why LLVM bitcode? +----------------- + +The [LLVM bitstream](http://llvm.org/docs/BitCodeFormat.html) format was +invented as a container format for LLVM IR. It is a binary format +supporting two basic structures: *blocks,* which define regions of the +file, and *records,* which contain data fields that can be up to 64 +bits. It has a few nice properties that make it a useful container +format for Swift modules as well: + +- It is easy to skip over an entire block, because the block's length + is recorded at its start. +- It is possible to jump to specific offsets *within* a block without + having to reparse from the start of the block. +- A format change doesn't immediately invalidate existing bitstream + files, because the stream includes layout information for + each record. +- It's a binary format, so it's at least *somewhat* compact. \[I + haven't done a size comparison against other formats.\] + +If we were to switch to another container format, we would likely want +it to have most of these properties as well. But we're already linking +against LLVM...might as well use it! + +Versioning +---------- + +> **warning** +> +> This section is relevant to any forward-compatible format used for a +> library's public interface. However, as mentioned above this may not +> be the current binary serialization format. +> +> Today's Swift uses a "major" version number of 0 and an +> always-incrementing "minor" version number. Every change is treated as +> compatibility-breaking; the minor version must match exactly for the +> compiler to load the module. + +Persistent serialized Swift files use the following versioning scheme: + +- Serialized modules are given a major and minor version number. +- When making a backwards-compatible change, the major and the minor + version number both MUST NOT be incremented. +- When making a change such that new modules cannot be safely loaded + by older compilers, the minor version number MUST be incremented. +- When making a change such that *old* modules cannot be safely loaded + by *newer* compilers, the major version number MUST be incremented. + The minor version number MUST then be reset to zero. +- Ideally, the major version number is never incremented. + +A serialized file's version number is checked against the client's +supported version before it is loaded. If it is too old or too new, the +file cannot be loaded. + +Note that the version number describes the contents of the file. Thus, +if a compiler supports features introduced in file version 1.9, but a +particular module only uses features introduced before and in version +1.7, the compiler MAY serialize that module with the version number 1.7. +However, doing so requires extra work on the compiler's part to detect +which features are in use; a simpler implementation would just use the +latest version number supported: 1.9. + +*This versioning scheme was inspired by* [Semantic +Versioning](http://semver.org). *However, it is not compatible with +Semantic Versioning because it promises* forward-compatibility *rather +than* backward-compatibility. + +A High-Level Tour of the Current Module Format +---------------------------------------------- + +Every serialized module is represented as a single block called the +"module block". The module block is made up of several other block +kinds, largely for organizational purposes. + +- The **block info block** is a standard LLVM bitcode block that + contains metadata about the bitcode stream. It is the only block + that appears outside the module block; we always put it at the very + start of the file. Though it can contain actual semantic + information, our use of it is only for debugging purposes. +- The **control block** is always the first block in the module block. + It can be processed without loading the rest of the module, and + indeed is intended to allow clients to decide whether not the module + is compatible with the current AST context. The major and minor + version numbers of the format are stored here. +- The **input block** contains information about how to import the + module once the client has decided to load it. This includes the + list of other modules that this module depends on. +- The **SIL block** contains SIL-level implementations that can be + imported into a client's SILModule context. In most cases this is + just a performance concern, but sometimes it affects language + semantics as well, as in the case of `@_transparent`. The SIL block + precedes the AST block because it affects which AST nodes + get serialized. +- The **SIL index black** contains tables for accessing various SIL + entities by their names, along with a mapping of unique IDs for + these to the appropriate bit offsets into the SIL block. +- The **AST block** contains the serialized forms of Decl, + DeclContext, and Type AST nodes. Decl nodes may be cross-references + to other modules, while types are always serialized with enough info + to regenerate them at load time. Nodes are accessed by a file-unique + "DeclIDs" (also covering DeclContexts) and "TypeIDs"; the two sets + of IDs use separate numbering schemes. + +> **note** +> +> The AST block is currently referred to as the "decls block" in the +> source. + +- The **identifier block** contains a single blob of strings. This is + intended for Identifiers---strings uniqued by the ASTContext---but + can in theory support any string data. The strings are accessed by a + file-unique "IdentifierID". +- The **index block** contains mappings from the AST node and + identifier IDs to their offsets in the AST block or identifier block + (as appropriate). It also contains various top-level AST information + about the module, such as its top-level declarations. + +SIL +--- + +\[to be written\] + +Cross-reference resilience +-------------------------- + +\[to be written\] diff --git a/docs/Serialization.rst b/docs/Serialization.rst deleted file mode 100644 index 9655ecf53a3d1..0000000000000 --- a/docs/Serialization.rst +++ /dev/null @@ -1,172 +0,0 @@ -:orphan: - -================================= -Swift Binary Serialization Format -================================= - -The fundamental unit of distribution for Swift code is a *module.* A module -contains declarations as an interface for clients to write code against. It may -also contain implementation information for any of these declarations that can -be used to optimize client code. Conceptually, the file containing the -interface for a module serves much the same purpose as the collection of C -header files for a particular library. - -Swift's binary serialization format is currently used for several purposes: - -- The public interface for a module ("swiftmodule files"). - -- A representation of captured compiler state after semantic analysis and SIL - generation, but before LLVM IR generation ("SIB", for "Swift Intermediate - Binary"). - -- Debug information about types, for proper high-level introspection without - running code. - -- Debug information about non-public APIs, for interactive debugging. - -The first two uses require a module to serve as a container of both AST nodes -and SIL entities. As a unit of distribution, it should also be -forward-compatible: module files installed on a developer's system in 201X -should be usable without updates for years to come, even as the Swift compiler -continues to be improved and enhanced. However, they are currently too closely -tied to the compiler internals to be useful for this purpose, and it is likely -we'll invent a new format instead. - - -Why LLVM bitcode? -================= - -The `LLVM bitstream `_ format was -invented as a container format for LLVM IR. It is a binary format supporting -two basic structures: *blocks,* which define regions of the file, and -*records,* which contain data fields that can be up to 64 bits. It has a few -nice properties that make it a useful container format for Swift modules as -well: - -- It is easy to skip over an entire block, because the block's length is - recorded at its start. - -- It is possible to jump to specific offsets *within* a block without having to - reparse from the start of the block. - -- A format change doesn't immediately invalidate existing bitstream files, - because the stream includes layout information for each record. - -- It's a binary format, so it's at least *somewhat* compact. [I haven't done a - size comparison against other formats.] - -If we were to switch to another container format, we would likely want it to -have most of these properties as well. But we're already linking against -LLVM...might as well use it! - - -Versioning -========== - -.. warning:: - - This section is relevant to any forward-compatible format used for a - library's public interface. However, as mentioned above this may not be - the current binary serialization format. - - Today's Swift uses a "major" version number of 0 and an always-incrementing - "minor" version number. Every change is treated as compatibility-breaking; - the minor version must match exactly for the compiler to load the module. - -Persistent serialized Swift files use the following versioning scheme: - -- Serialized modules are given a major and minor version number. - -- When making a backwards-compatible change, the major and the minor version - number both MUST NOT be incremented. - -- When making a change such that new modules cannot be safely loaded by older - compilers, the minor version number MUST be incremented. - -- When making a change such that *old* modules cannot be safely loaded by - *newer* compilers, the major version number MUST be incremented. The minor - version number MUST then be reset to zero. - -- Ideally, the major version number is never incremented. - -A serialized file's version number is checked against the client's supported -version before it is loaded. If it is too old or too new, the file cannot be -loaded. - -Note that the version number describes the contents of the file. Thus, if a -compiler supports features introduced in file version 1.9, but a particular -module only uses features introduced before and in version 1.7, the compiler -MAY serialize that module with the version number 1.7. However, doing so -requires extra work on the compiler's part to detect which features are in use; -a simpler implementation would just use the latest version number supported: -1.9. - -*This versioning scheme was inspired by* `Semantic Versioning -`_. *However, it is not compatible with Semantic Versioning -because it promises* forward-compatibility *rather than* backward-compatibility. - - -A High-Level Tour of the Current Module Format -============================================== - -Every serialized module is represented as a single block called the "module -block". The module block is made up of several other block kinds, largely for -organizational purposes. - -- The **block info block** is a standard LLVM bitcode block that contains - metadata about the bitcode stream. It is the only block that appears outside - the module block; we always put it at the very start of the file. Though it - can contain actual semantic information, our use of it is only for debugging - purposes. - -- The **control block** is always the first block in the module block. It can - be processed without loading the rest of the module, and indeed is intended - to allow clients to decide whether not the module is compatible with the - current AST context. The major and minor version numbers of the format are - stored here. - -- The **input block** contains information about how to import the module once - the client has decided to load it. This includes the list of other modules - that this module depends on. - -- The **SIL block** contains SIL-level implementations that can be imported - into a client's SILModule context. In most cases this is just a performance - concern, but sometimes it affects language semantics as well, as in the case - of ``@_transparent``. The SIL block precedes the AST block because it affects - which AST nodes get serialized. - -- The **SIL index black** contains tables for accessing various SIL entities by - their names, along with a mapping of unique IDs for these to the appropriate - bit offsets into the SIL block. - -- The **AST block** contains the serialized forms of Decl, DeclContext, and - Type AST nodes. Decl nodes may be cross-references to other modules, while - types are always serialized with enough info to regenerate them at load time. - Nodes are accessed by a file-unique "DeclIDs" (also covering DeclContexts) - and "TypeIDs"; the two sets of IDs use separate numbering schemes. - -.. note:: - - The AST block is currently referred to as the "decls block" in the source. - -- The **identifier block** contains a single blob of strings. This is intended - for Identifiers---strings uniqued by the ASTContext---but can in theory - support any string data. The strings are accessed by a file-unique - "IdentifierID". - -- The **index block** contains mappings from the AST node and identifier IDs to - their offsets in the AST block or identifier block (as appropriate). It also - contains various top-level AST information about the module, such as its - top-level declarations. - - -SIL -=== - -[to be written] - - -Cross-reference resilience -========================== - -[to be written] diff --git a/docs/StdlibAPIGuidelines.md b/docs/StdlibAPIGuidelines.md new file mode 100644 index 0000000000000..2431b003ad5e5 --- /dev/null +++ b/docs/StdlibAPIGuidelines.md @@ -0,0 +1,139 @@ +Swift Standard Library API Design Guide +======================================= + +> **note** +> +> This guide documents *current practice* in the Swift +> +> : standard library as of April 2015. API conventions are expected to +> evolve in the near future to better harmonize with Cocoa. +> +The current Swift Standard Library API conventions start with the Cocoa +guidelines as discussed on these two wiki pages: \[[API +Guidelines](http://cocoa.apple.com/cgi-bin/wiki.pl?API_Guidelines), +[Properties](http://cocoa.apple.com/cgi-bin/wiki.pl?Properties)\], and +in this [WWDC Presentation](http://cocoa.apple.com/CocoaAPIDesign.pdf). +Below, we list where and how the standard library's API conventions +differ from those of Cocoa + +Differences +----------- + +Points in this section clash in one way or other with the Cocoa +guidelines. + +### The First Parameter + +- The first parameter to a function, method, or initializer typically + does not have an argument label: +- Typically, no suffix is added to a function or method's base name in + order to serve the same purpose as a label: +- A preposition is added to the end of a function name if the role of + the first parameter would otherwise be unclear: +- Argument labels are used on first parameters to denote special + cases: + +### Subsequent Parameters + +- Argument labels are chosen to clarify the *role* of an argument, + rather than its type: +- Second and later parameters are always labeled except in cases where + there's no useful distinction of roles: + + swap(&a, &b) // OK + + let topOfPicture = min(topOfSquare, topOfTriangle, topOfCircle) // OK + +### Other Differences + +- We don't use namespace prefixes such as “`NS`{.sourceCode}”, relying + instead on the language's own facilities. +- Names of types, protocols and enum cases are + `UpperCamelCase`{.sourceCode}. Everything else is + `lowerCamelCase`{.sourceCode}. When an initialism appears, it is + **uniformly upper- or lower-cased to fit the pattern**: +- Protocol names end in `Type`{.sourceCode}, `able`{.sourceCode}, or + `ible`{.sourceCode}. Other type names do not. + +Additional Conventions +---------------------- + +Points in this section place additional constraints on the standard +library, but are compatible with the Cocoa guidelines. + +- We document the complexity of operations using big-O notation. +- In API design, when deciding between a nullary function and a + property for a specific operation, arguments based on performance + characteristics and complexity of operations are not considered. + Reading and writing properties can have any complexity. +- We prefer methods and properties to free functions. Free functions + are used when there's no obvious `self`{.sourceCode} : + + min(x, y, z) + + when the function is an unconstrained generic : + + print(x) + + and when function syntax is part of the domain notation : + + -sin(x) + +- Type conversions use initialization syntax whenever possible, with + the source of the conversion being the first argument: + + let s0 = String(anInt) // yes + let s1 = String(anInt, radix: 2) // yes + let s1 = anInt.toString() // no + + The exception is when the type conversion is part of a protocol: + + protocol IntConvertible { + func toInt() -> Int // OK + } + +- Even unlabelled parameter names should be meaningful as they'll be + referred to in comments and visible in “generated headers” + (cmd-click in Xcode): +- Type parameter names of generic types describe the role of the + parameter, e.g. + +### Acceptable Short or Non-Descriptive Names + +- Type parameter names of generic functions may be single characters: +- `lhs`{.sourceCode} and `rhs`{.sourceCode} are acceptable names for + binary operator or symmetric binary function parameters: +- `body`{.sourceCode} is an acceptable name for a trailing closure + argument when the resulting construct is supposed to act like a + language extension and is likely to have side-effects: + + func map(transformation: T->U) -> [U] // not this one + + func forEach(body: (S.Generator.Element)->()) + +### Prefixes and Suffixes + +- `Any`{.sourceCode} is used as a prefix to denote “type + erasure,” e.g. `AnySequence`{.sourceCode} wraps any sequence with + element type `T`{.sourceCode}, conforms to + `SequenceType`{.sourceCode} itself, and forwards all operations to + the wrapped sequence. When handling the wrapper, the specific type + of the wrapped sequence is fully hidden. +- `Custom`{.sourceCode} is used as a prefix for special protocols that + will always be dynamically checked for at runtime and don't make + good generic constraints, e.g. + `CustomStringConvertible`{.sourceCode}. +- `InPlace`{.sourceCode} is used as a suffix to denote the mutating + member of a pair of related methods: +- `with`{.sourceCode} is used as a prefix to denote a function that + executes a closure within a context, such as a guaranteed lifetime: +- `Pointer`{.sourceCode} is used as a suffix to denote a non-class + type that acts like a reference, c.f. + `ManagedBufferPointer`{.sourceCode} +- `unsafe`{.sourceCode} or `Unsafe`{.sourceCode} is *always* used as a + prefix when a function or type allows the user to violate memory or + type safety, except on methods of types whose names begin with + `Unsafe`{.sourceCode}, where the type name is assumed to + convey that. +- `C`{.sourceCode} is used as a prefix to denote types corresponding + to C language types, e.g. `CChar`{.sourceCode}. diff --git a/docs/StdlibAPIGuidelines.rst b/docs/StdlibAPIGuidelines.rst deleted file mode 100644 index 26004130fac61..0000000000000 --- a/docs/StdlibAPIGuidelines.rst +++ /dev/null @@ -1,247 +0,0 @@ -:orphan: - -.. default-role:: code - -======================================= -Swift Standard Library API Design Guide -======================================= - -.. Note:: This guide documents *current practice* in the Swift - standard library as of April 2015. API conventions are - expected to evolve in the near future to better harmonize - with Cocoa. - -The current Swift Standard Library API conventions start with the -Cocoa guidelines as discussed on these two wiki pages: [`API -Guidelines `_, -`Properties `_], -and in this `WWDC Presentation -`_. Below, we list where -and how the standard library's API conventions differ from those of -Cocoa - -Differences -=========== - -Points in this section clash in one way or other with the Cocoa -guidelines. - -The First Parameter -------------------- - -* The first parameter to a function, method, or initializer typically - does not have an argument label: - - .. parsed-literal:: - - alligators.insert(fred) // yes - if alligators.contains(george) { // yes - return - } - - alligators.insert(**element:** fred) // no - if alligators.contains(**element:** george) { // no - return - } - -* Typically, no suffix is added to a function or method's base name in - order to serve the same purpose as a label: - - .. parsed-literal:: - - alligators.insert\ **Element**\ (fred) // no - if alligators.contains\ **Element**\ (george) { // no - return - } - - -* A preposition is added to the end of a function name if the role of - the first parameter would otherwise be unclear: - - .. parsed-literal:: - - // origin of measurement is aPosition - aPosition.distance\ **To**\ (otherPosition) - - // we're not "indexing x" - if let position = aSet.index\ **Of**\ (x) { ... } - -* Argument labels are used on first parameters to denote special - cases: - - .. parsed-literal:: - - // Normal case: result has same value as argument (traps on overflow) - Int(aUInt) - - // Special: interprets the sign bit as a high bit, changes value - Int(**bitPattern**: aUInt) - - // Special: keeps only the bits that fit, losing information - Int32(**truncatingBitPattern**: anInt64) - -Subsequent Parameters ---------------------- - -* Argument labels are chosen to clarify the *role* of an argument, - rather than its type: - - .. parsed-literal:: - - x.replaceRange(r, **with:** someElements) - - p.initializeFrom(q, **count:** n) - -* Second and later parameters are always labeled except in cases where - there's no useful distinction of roles:: - - swap(&a, &b) // OK - - let topOfPicture = min(topOfSquare, topOfTriangle, topOfCircle) // OK - -Other Differences ------------------ - -* We don't use namespace prefixes such as “`NS`”, relying instead on - the language's own facilities. - -* Names of types, protocols and enum cases are `UpperCamelCase`. - Everything else is `lowerCamelCase`. When an initialism appears, it - is **uniformly upper- or lower-cased to fit the pattern**: - - .. parsed-literal:: - - let v: String.\ **UTF**\ 16View = s.\ **utf**\ 16 - -* Protocol names end in `Type`, `able`, or `ible`. Other type names - do not. - -Additional Conventions -====================== - -Points in this section place additional constraints on the standard -library, but are compatible with the Cocoa guidelines. - -* We document the complexity of operations using big-O notation. - -* In API design, when deciding between a nullary function and a property for a - specific operation, arguments based on performance characteristics and - complexity of operations are not considered. Reading and writing properties - can have any complexity. - -* We prefer methods and properties to free functions. Free functions - are used when there's no obvious `self` :: - - min(x, y, z) - - when the function is an unconstrained generic :: - - print(x) - - and when function syntax is part of the domain notation :: - - -sin(x) - -* Type conversions use initialization syntax whenever possible, with - the source of the conversion being the first argument:: - - let s0 = String(anInt) // yes - let s1 = String(anInt, radix: 2) // yes - let s1 = anInt.toString() // no - - The exception is when the type conversion is part of a protocol:: - - protocol IntConvertible { - func toInt() -> Int // OK - } - -* Even unlabelled parameter names should be meaningful as they'll be - referred to in comments and visible in “generated headers” - (cmd-click in Xcode): - - .. parsed-literal:: - - /// Reserve enough space to store \`\ **minimumCapacity**\ \` elements. - /// - /// PostCondition: \`\ capacity >= **minimumCapacity**\ \` and the array has - /// mutable contiguous storage. - /// - /// Complexity: O(\`count\`) - mutating func reserveCapacity(**minimumCapacity**: Int) - -* Type parameter names of generic types describe the role of the - parameter, e.g. - - .. parsed-literal:: - - struct Dictionary<**Key**, **Value**> { // *not* Dictionary<**K**, **V**> - -Acceptable Short or Non-Descriptive Names ------------------------------------------ - -* Type parameter names of generic functions may be single characters: - - .. parsed-literal:: - - func swap<**T**>(inout lhs: T, inout rhs: T) - -* `lhs` and `rhs` are acceptable names for binary operator or - symmetric binary function parameters: - - .. parsed-literal:: - - func + (**lhs**: Int, **rhs**: Int) -> Int - - func swap(inout **lhs**: T, inout **rhs**: T) - -* `body` is an acceptable name for a trailing closure argument when - the resulting construct is supposed to act like a language extension - and is likely to have side-effects:: - - func map(transformation: T->U) -> [U] // not this one - - func forEach(body: (S.Generator.Element)->()) - -Prefixes and Suffixes ---------------------- - -* `Any` is used as a prefix to denote “type erasure,” - e.g. `AnySequence` wraps any sequence with element type `T`, - conforms to `SequenceType` itself, and forwards all operations to the - wrapped sequence. When handling the wrapper, the specific type of - the wrapped sequence is fully hidden. - -* `Custom` is used as a prefix for special protocols that will always - be dynamically checked for at runtime and don't make good generic - constraints, e.g. `CustomStringConvertible`. - -* `InPlace` is used as a suffix to denote the mutating member of a - pair of related methods: - - .. parsed-literal:: - - extension Set { - func union(other: Set) -> Set - mutating func union\ **InPlace**\ (other: Set) - } - -* `with` is used as a prefix to denote a function that executes a - closure within a context, such as a guaranteed lifetime: - - .. parsed-literal:: - - s.\ **with**\ CString { - let fd = fopen($0) - ... - } // don't use that pointer after the closing brace - -* `Pointer` is used as a suffix to denote a non-class type that acts - like a reference, c.f. `ManagedBufferPointer` - -* `unsafe` or `Unsafe` is *always* used as a prefix when a function or - type allows the user to violate memory or type safety, except on - methods of types whose names begin with `Unsafe`, where the type - name is assumed to convey that. - -* `C` is used as a prefix to denote types corresponding to C language - types, e.g. `CChar`. diff --git a/docs/StdlibRationales.md b/docs/StdlibRationales.md new file mode 100644 index 0000000000000..9a9d76e71535e --- /dev/null +++ b/docs/StdlibRationales.md @@ -0,0 +1,162 @@ +Rationales for the Swift standard library designs +================================================= + +This document collects rationales for the Swift standard library. It is +not meant to document all possible designs that we considered, but might +describe some of those, when important to explain the design that was +chosen. + +Current designs +--------------- + +### Some `NSString` APIs are mirrored on `String` + +There was not enough time in Swift 1.0 to design a rich `String` API, so +we reimplemented most of `NSString` APIs on `String` for parity. This +brought the exact `NSString` semantics of those APIs, for example, +treatment of Unicode or behavior in edge cases (for example, empty +strings), which we might want to reconsider. + +Radars: rdar://problem/19705854 + +### `size_t` is unsigned, but it is imported as `Int` + +Converging APIs to use `Int` as the default integer type allows users to +write fewer explicit type conversions. + +Importing `size_t` as a signed `Int` type would not be a problem for +64-bit platforms. The only concern is about 32-bit platforms, and only +about operating on array-like data structures that span more than half +of the address space. Even today, in 2015, there are enough 32-bit +platforms that are still interesting, and x32 ABIs for 64-bit CPUs are +also important. We agree that 32-bit platforms are important, but the +usecase for an unsigned `size_t` on 32-bit platforms is pretty marginal, +and for code that nevertheless needs to do that there is always the +option of doing a bitcast to `UInt` or using C. + +### Type Conversions + +The canonical way to convert from an instance x of type `T` to type `U` +is `U(x)`, a precedent set by `Int(value: UInt32)`. Conversions that can +fail should use failable initializers, e.g. `Int(text: String)`, +yielding a `Int?`. When other forms provide added convenience, they may +be provided as well. For example: + + String.Index(s.utf16.startIndex.successor(), within: s) // canonical + s.utf16.startIndex.successor().samePositionIn(s) // alternate + +Converting initializers generally take one parameter. A converting +initializer's first parameter should not have an argument label unless +it indicates a lossy, non-typesafe, or non-standard conversion method, +e.g. `Int(bitPattern: someUInt)`. When a converting initializer requires +a parameter for context, it should not come first, and generally +*should* use a keyword. For example, `String(33, radix: 2)`. + +Rationale + +: First, type conversions are typical trouble spots, and we like the + idea that people are explicit about the types to which + they're converting. Secondly, avoiding method or property syntax + provides a distinct context for code completion. Rather than + appearing in completions offered after `.`, for example, the + available conversions could show up whenever the user hit the “tab” + key after an expression. + +### Protocols with restricted conformance rules + +It is sometimes useful to define a public protocol that only a limited +set of types can adopt. There is no language feature in Swift to +disallow declaring conformances in third-party code: as long as the +requirements are implemented and the protocol is accessible, the +compiler allows the conformance. + +The standard library adopts the following pattern: the protocol is +declared as a regular public type, but it includes at least one +requirement named using the underscore rule. That underscored API +becomes private to the users according to the standard library +convention, effectively preventing third-party code from declaring a +conformance. + +For example: + + public protocol CVarArgType { + var _cVarArgEncoding: [Word] { get } + } + + // Public API that uses CVaListPointer, so CVarArgType has to be public, too. + public func withVaList( + args: [CVarArgType], + @noescape f: (CVaListPointer) -> R + ) -> R + +### High-order functions on collections return `Array`s + +We can't make `map()`, `filter()`, etc. all return `Self`: + +- `map()` takes a function `(T) -> U` and therefore can't return + Self literally. The required language feature for making `map()` + return something like `Self` in generic code (higher-kinded types) + doesn't exist in Swift. You can't write a method like + `func map(f: (T) -> U) -> Self` today. +- There are lots of sequences that don't have an appropriate form for + the result. What happens when you filter the only element out of a + `SequenceOfOne`, which is defined to have exactly one element? +- A `map()` that returns `Self` hews most closely to the signature + required by Functor (mathematical purity of signature), but if you + make map on `Set` or `Dictionary` return `Self`, it violates the + semantic laws required by Functor, so it's a false purity. We'd + rather preserve the semantics of functional `map()` than + its signature. +- The behavior is surprising (and error-prone) in generic code: + + func countFlattenedElements< + S : SequenceType where S.Generator.Element == Set + >(sequence: S) -> Int { + return sequence.map { $0.count }.reduce(0) { $0 + $1 } + } + +The function behaves as expected when given an `[Set]`, but the +results are wrong for `Set>`. The `sequence.map()` operation +would return a `Set`, and all non-unique counts would disappear. + +- Even if we throw semantics under the bus, maintaining mathematical + purity of signature prevents us from providing useful variants of + these algorithms that are the same in spirit, like the `flatMap()` + that selects the non-nil elements of the result sequence. + +### The remove\*() method family on collections + +Protocol extensions for `RangeReplaceableCollectionType` define +`removeFirst(n: Int)` and `removeLast(n: Int)`. These functions remove +exactly `n` elements; they don't clamp `n` to `count` or they could be +masking bugs. + +Since the standard library tries to preserve information, it also +defines special overloads that return just one element, +`removeFirst() -> Element` and `removeLast() -> Element`, that return +the removed element. These overloads have a precondition that the +collection is not empty. Another possible design would be that they +don't have preconditions and return `Element?`. Doing so would make the +overload set inconsistent: semantics of different overloads would be +significantly different. It would be surprising that +`myData.removeFirst()` and `myData.removeFirst(1)` are not equivalent. + +### Lazy functions that operate on sequences and collections + +In many cases functions that operate on sequences can be implemented +either lazily or eagerly without compromising performance. To decide +between a lazy and an eager implementation, the standard library uses +the following rule. When there is a choice, and not explicitly required +by the API semantics, functions don't return lazy collection wrappers +that refer to users' closures. The consequence is that all users' +closures are `@noescape`, except in an explicitly lazy context. + +Based on this rule, we conclude that +`enumerate(),`zip()`and`reverse()`return lazy wrappers, but`filter()`and`map()`don't. For the first three functions being lazy is the right default, since usually the result is immediately consumed by for-in, so we don't want to allocate memory for it. A different design that was rejected is to preserve consistency with other strict functions by making these methods strict, but then client code needs to call an API with a different name, say`lazyEnumerate()`to opt into laziness. The problem is that the eager API, which would have a shorter and less obscure name, would be less efficient for the common case. Use of`BooleanType`in library APIs -------------------------------------- Use`Bool`instead of a generic function over a`BooleanType`, unless there are special circumstances (for example,`func +&&`is designed to work on all boolean values so that`&&`feels like a part of the language).`BooleanType`is a protocol to which only`Bool`and`ObjCBool`conform. Users don't usually interact`ObjCBool`instances, except when using certain specific APIs (for example, APIs that operate on pointers to`BOOL`). If someone already has an`ObjCBool`instance for whatever strange reason, they can just convert it to`Bool`. We think this is the right tradeoff: simplifying function signatures is more important than making a marginal usecase a bit more convenient. Possible future directions ========================== This section describes some of the possible future designs that we have discussed. Some might get dismissed, others might become full proposals and get implemented. Mixed-type fixed-point arithmetic --------------------------------- Radars: rdar://problem/18812545 rdar://problem/18812365 Standard library only defines arithmetic operators for LHS and RHS that have matching types. It might be useful to allow users to mix types. There are multiple approaches: * AIR model, * overloads in the standard library for operations that are always safe and can't trap (e.g., comparisons), * overloads in the standard library for all operations. TODO: describe advantages The arguments towards not doing any of these, at least in the short term: * demand might be lower than we think: seems like users have converged towards using`Int`as the default integer type. * mitigation: import good C APIs that use appropriate typedefs for unsigned integers (`size\_t`for example) as`Int`. Swift: Power operator --------------------- Radars: rdar://problem/17283778 It would be very useful to have a power operator in Swift. We want to make code look as close as possible to the domain notation, the two-dimensional formula in this case. In the two-dimensional representation exponentiation is represented by a change in formatting. With`pow()`, once you see the comma, you have to scan to the left and count parentheses to even understand that there is a`pow()`there. The biggest concern is that adding an operator has a high barrier. Nevertheless, we agree`\*\*\`\` +is the right way to spell it, if we were to have it. Also there was some +agreement that if we did not put this operator in the core library (so +that you won't get it by default), it would become much more compelling. + +We will revisit the discussion when we have submodules for the standard +library, in one form or the other. diff --git a/docs/StdlibRationales.rst b/docs/StdlibRationales.rst deleted file mode 100644 index 02d423d4b0a6d..0000000000000 --- a/docs/StdlibRationales.rst +++ /dev/null @@ -1,244 +0,0 @@ -:orphan: - -.. @raise litre.TestsAreMissing - -================================================= -Rationales for the Swift standard library designs -================================================= - -This document collects rationales for the Swift standard library. It is not -meant to document all possible designs that we considered, but might describe -some of those, when important to explain the design that was chosen. - -Current designs -=============== - -Some ``NSString`` APIs are mirrored on ``String`` -------------------------------------------------- - -There was not enough time in Swift 1.0 to design a rich ``String`` API, so we -reimplemented most of ``NSString`` APIs on ``String`` for parity. This brought -the exact ``NSString`` semantics of those APIs, for example, treatment of -Unicode or behavior in edge cases (for example, empty strings), which we might -want to reconsider. - -Radars: rdar://problem/19705854 - -``size_t`` is unsigned, but it is imported as ``Int`` ------------------------------------------------------ - -Converging APIs to use ``Int`` as the default integer type allows users to -write fewer explicit type conversions. - -Importing ``size_t`` as a signed ``Int`` type would not be a problem for 64-bit -platforms. The only concern is about 32-bit platforms, and only about -operating on array-like data structures that span more than half of the address -space. Even today, in 2015, there are enough 32-bit platforms that are still -interesting, and x32 ABIs for 64-bit CPUs are also important. We agree that -32-bit platforms are important, but the usecase for an unsigned ``size_t`` on -32-bit platforms is pretty marginal, and for code that nevertheless needs to do -that there is always the option of doing a bitcast to ``UInt`` or using C. - -Type Conversions ----------------- - -The canonical way to convert from an instance `x` of type ``T`` to -type ``U`` is ``U(x)``, a precedent set by ``Int(value: UInt32)``. -Conversions that can fail should use failable initializers, -e.g. ``Int(text: String)``, yielding a ``Int?``. When other forms provide -added convenience, they may be provided as well. For example:: - - String.Index(s.utf16.startIndex.successor(), within: s) // canonical - s.utf16.startIndex.successor().samePositionIn(s) // alternate - -Converting initializers generally take one parameter. A converting -initializer's first parameter should not have an argument label unless -it indicates a lossy, non-typesafe, or non-standard conversion method, -e.g. ``Int(bitPattern: someUInt)``. When a converting initializer -requires a parameter for context, it should not come first, and -generally *should* use a keyword. For example, ``String(33, radix: -2)``. - -:Rationale: First, type conversions are typical trouble spots, and we - like the idea that people are explicit about the types to which - they're converting. Secondly, avoiding method or property syntax - provides a distinct context for code completion. Rather than - appearing in completions offered after ``.``, for example, the - available conversions could show up whenever the user hit the “tab” - key after an expression. - -Protocols with restricted conformance rules -------------------------------------------- - -It is sometimes useful to define a public protocol that only a limited set of -types can adopt. There is no language feature in Swift to disallow declaring -conformances in third-party code: as long as the requirements are implemented -and the protocol is accessible, the compiler allows the conformance. - -The standard library adopts the following pattern: the protocol is declared as -a regular public type, but it includes at least one requirement named using the -underscore rule. That underscored API becomes private to the users according -to the standard library convention, effectively preventing third-party code from -declaring a conformance. - -For example:: - - public protocol CVarArgType { - var _cVarArgEncoding: [Word] { get } - } - - // Public API that uses CVaListPointer, so CVarArgType has to be public, too. - public func withVaList( - args: [CVarArgType], - @noescape f: (CVaListPointer) -> R - ) -> R - -High-order functions on collections return ``Array``\ s -------------------------------------------------------- - -We can't make ``map()``, ``filter()``, etc. all return ``Self``: - -- ``map()`` takes a function ``(T) -> U`` and therefore can't return Self - literally. The required language feature for making ``map()`` return - something like ``Self`` in generic code (higher-kinded types) doesn't exist - in Swift. You can't write a method like ``func map(f: (T) -> U) -> Self`` - today. - -- There are lots of sequences that don't have an appropriate form for the - result. What happens when you filter the only element out of a - ``SequenceOfOne``, which is defined to have exactly one element? - -- A ``map()`` that returns ``Self`` hews most closely to the signature - required by Functor (mathematical purity of signature), but if you make map - on ``Set`` or ``Dictionary`` return ``Self``, it violates the semantic laws - required by Functor, so it's a false purity. We'd rather preserve the - semantics of functional ``map()`` than its signature. - -- The behavior is surprising (and error-prone) in generic code:: - - func countFlattenedElements< - S : SequenceType where S.Generator.Element == Set - >(sequence: S) -> Int { - return sequence.map { $0.count }.reduce(0) { $0 + $1 } - } - -The function behaves as expected when given an ``[Set]``, but the -results are wrong for ``Set>``. The ``sequence.map()`` operation -would return a ``Set``, and all non-unique counts would disappear. - -- Even if we throw semantics under the bus, maintaining mathematical purity of - signature prevents us from providing useful variants of these algorithms that - are the same in spirit, like the ``flatMap()`` that selects the non-nil - elements of the result sequence. - -The `remove*()` method family on collections --------------------------------------------- - -Protocol extensions for ``RangeReplaceableCollectionType`` define -``removeFirst(n: Int)`` and ``removeLast(n: Int)``. These functions remove -exactly ``n`` elements; they don't clamp ``n`` to ``count`` or they could be -masking bugs. - -Since the standard library tries to preserve information, it also defines -special overloads that return just one element, ``removeFirst() -> Element`` -and ``removeLast() -> Element``, that return the removed element. These -overloads have a precondition that the collection is not empty. Another -possible design would be that they don't have preconditions and return -``Element?``. Doing so would make the overload set inconsistent: semantics of -different overloads would be significantly different. It would be surprising -that ``myData.removeFirst()`` and ``myData.removeFirst(1)`` are not equivalent. - -Lazy functions that operate on sequences and collections --------------------------------------------------------- - -In many cases functions that operate on sequences can be implemented either -lazily or eagerly without compromising performance. To decide between a lazy -and an eager implementation, the standard library uses the following rule. -When there is a choice, and not explicitly required by the API semantics, -functions don't return lazy collection wrappers that refer to users' closures. -The consequence is that all users' closures are ``@noescape``, except in an -explicitly lazy context. - -Based on this rule, we conclude that ``enumerate(), ``zip()`` and -``reverse()`` return lazy wrappers, but ``filter()`` and ``map()`` don't. For -the first three functions being lazy is the right default, since usually the -result is immediately consumed by for-in, so we don't want to allocate memory -for it. - -A different design that was rejected is to preserve consistency with other -strict functions by making these methods strict, but then client code needs to -call an API with a different name, say ``lazyEnumerate()`` to opt into -laziness. The problem is that the eager API, which would have a shorter and -less obscure name, would be less efficient for the common case. - -Use of ``BooleanType`` in library APIs --------------------------------------- - -Use ``Bool`` instead of a generic function over a ``BooleanType``, unless there -are special circumstances (for example, ``func &&`` is designed to work on all -boolean values so that ``&&`` feels like a part of the language). - -``BooleanType`` is a protocol to which only ``Bool`` and ``ObjCBool`` conform. -Users don't usually interact ``ObjCBool`` instances, except when using certain -specific APIs (for example, APIs that operate on pointers to ``BOOL``). If -someone already has an ``ObjCBool`` instance for whatever strange reason, they -can just convert it to ``Bool``. We think this is the right tradeoff: -simplifying function signatures is more important than making a marginal -usecase a bit more convenient. - -Possible future directions -========================== - -This section describes some of the possible future designs that we have -discussed. Some might get dismissed, others might become full proposals and -get implemented. - -Mixed-type fixed-point arithmetic ---------------------------------- - -Radars: rdar://problem/18812545 rdar://problem/18812365 - -Standard library only defines arithmetic operators for LHS and RHS that have -matching types. It might be useful to allow users to mix types. - -There are multiple approaches: - -* AIR model, - -* overloads in the standard library for operations that are always safe and - can't trap (e.g., comparisons), - -* overloads in the standard library for all operations. - -TODO: describe advantages - -The arguments towards not doing any of these, at least in the short term: - -* demand might be lower than we think: seems like users have converged towards - using ``Int`` as the default integer type. - -* mitigation: import good C APIs that use appropriate typedefs for - unsigned integers (``size_t`` for example) as ``Int``. - - -Swift: Power operator ---------------------- - -Radars: rdar://problem/17283778 - -It would be very useful to have a power operator in Swift. We want to make -code look as close as possible to the domain notation, the two-dimensional -formula in this case. In the two-dimensional representation exponentiation is -represented by a change in formatting. With ``pow()``, once you see the comma, -you have to scan to the left and count parentheses to even understand that -there is a ``pow()`` there. - -The biggest concern is that adding an operator has a high barrier. -Nevertheless, we agree ``**`` is the right way to spell it, if we were to have -it. Also there was some agreement that if we did not put this operator in the -core library (so that you won't get it by default), it would become much more -compelling. - -We will revisit the discussion when we have submodules for the standard -library, in one form or the other. - diff --git a/docs/StoredAndComputedVariables.md b/docs/StoredAndComputedVariables.md new file mode 100644 index 0000000000000..fbe513715cafb --- /dev/null +++ b/docs/StoredAndComputedVariables.md @@ -0,0 +1,286 @@ +Stored and Computed Variables +============================= + +> **warning** +> +> This document has not been updated since the initial design in + +> Swift 1.0. + +Variables are declared using the `var` keyword. These declarations are +valid at the top level, within types, and within code bodies, and are +respectively known as *global variables,* *member variables,* and *local +variables.* Member variables are commonly referred to as *properties.* + +Every variable declaration can be classified as either *stored* or +*computed.* Member variables inherited from a superclass obey slightly +different rules. + +Stored Variables +---------------- + +The simplest form of a variable declaration provides only a type: + + var count : Int + +This form of `var` declares a *stored variable.* Stored variables cause +storage to be allocated in their containing context: + +- a new global symbol for a global variable +- a slot in an object for a member variable +- space on the stack for a local variable + +(Note that this storage may still be optimized away if determined +unnecessary.) + +Stored variables must be initialized before use. As such, an initial +value can be provided at the declaration site. This is mandatory for +global variables, since it cannot be proven who accesses the variable +first. : + + var count : Int = 10 + +If the type of the variable can be inferred from the initial value +expression, it may be omitted in the declaration: + + var count = 10 + +Variables formed during pattern matching are also considered stored +variables. : + + switch optVal { + case .Some(var actualVal): + // do something + case .None: + // do something else + } + +Computed Variables +------------------ + +A *computed variable* behaves syntactically like a variable, but does +not actually require storage. Instead, accesses to the variable go +through "accessors" known as the *getter* and the *setter.* Thus, a +computed variable is declared as a variable with a custom getter: + + struct Rect { + // Stored member variables + var x, y, width, height : Int + + // A computed member variable + var maxX : Int { + get { + return x + width + } + set(newMax) { + x = newMax - width + } + } + + // myRect.maxX = 40 + +In this example, no storage is provided for `maxX`. + +If the setter's argument is omitted, it is assumed to be named `value`: + + var maxY : Int { + get { + return y + height + } + set { + y = value - height + } + } + +Finally, if a computed variable has a getter but no setter, it becomes a +*read-only variable.* In this case the `get` label may be omitted. +Attempting to set a read-only variable is a compile-time error: + + var area : Int { + return self.width * self.height + } + +> } + +Note that because this is a member variable, the implicit parameter +`self` is available for use within the accessors. + +It is illegal for a variable to have a setter but no getter. + +Observing Accessors +------------------- + +Occasionally it is useful to provide custom behavior when changing a +variable's value that goes beyond simply modifying the underlying +storage. One way to do this is to pair a stored variable with a computed +variable: + + var _backgroundColor : Color + var backgroundColor : Color { + get { + return _backgroundColor + } + set { + _backgroundColor = value + refresh() + } + } + +However, this contains a fair amount of boilerplate. For cases where a +stored property provides the correct storage semantics, you can add +custom behavior before or after the underlying assignment using +"observing accessors" `willSet` and `didSet`: + + var backgroundColor : Color { + didSet { + refresh() + } + } + + var currentURL : URL { + willSet(newValue) { + if newValue != currentURL { + cancelCurrentRequest() + } + } + didSet { + sendNewRequest(currentURL) + } + } + +A stored property may have either observing accessor, or both. Like +`set`, the argument for `willSet` may be omitted, in which case it is +provided as "value": + + var accountName : String { + willSet { + assert(value != "root") + } + } + +Observing accessors provide the same behavior as the two-variable +example, with two important exceptions: + +- A variable with observing accessors is still a stored variable, + which means it must still be initialized before use. Initialization + does not run the code in the observing accessors. +- All assignments to the variable will trigger the observing accessors + with the following exceptions: assignments in the init and + destructor function for the enclosing type, and those from within + the accessors themselves. In this context, assignments directly + store to the underlying storage. + +Computed properties may not have observing accessors. That is, a +property may have a custom getter or observing accessors, but not both. + +Overriding Read-Only Variables +------------------------------ + +If a member variable within a class is a read-only computed variable, it +may be overridden by subclasses. In this case, the subclass may choose +to replace that computed variable with a stored variable by declaring +the stored variable in the usual way: + + class Base { + var color : Color { + return .Black + } + } + + class Colorful : Base { + var color : Color + } + + var object = Colorful(.Red) + object.color = .Blue + +The new stored variable may have observing accessors: + + class MemoryColorful : Base { + var oldColors : Array = [] + + var color : Color { + willSet { + oldColors.append(color) + } + } + } + +A computed variable may also be overridden with another computed +variable: + + class MaybeColorful : Base { + var color : Color { + get { + if randomBooleanValue() { + return .Green + } else { + return super.color + } + } + set { + print("Sorry, we choose our own colors here.") + } + } + } + +Overriding Read-Write Variables +------------------------------- + +If a member variable within a class as a read-write variable, it is not +generally possible to know if it is a computed variable or stored +variable. A subclass may override the superclass's variable with a new +computed variable: + + class ColorBase { + var color : Color { + didSet { + print("I've been painted \(color)!") + } + } + } + + class BrightlyColored : ColorBase { + var color : Color { + get { + return super.color + } + set(newColor) { + // Prefer whichever color is brighter. + if newColor.luminance > super.color.luminance { + super.color = newColor + } else { + // Keep the old color. + } + } + } + } + +In this case, because the superclass's `didSet` is part of the generated +setter, it is only called when the subclass actually invokes setter +through its superclass. On the `else` branch, the superclass's `didSet` +is skipped. + +A subclass may also use observing accessors to add behavior to an +inherited member variable: + + class TrackingColored : ColorBase { + var prevColor : Color? + + var color : Color { + willSet { + prevColor = color + } + } + } + +In this case, the `willSet` accessor in the subclass is called first, +then the setter for `color` in the superclass. Critically, this is *not* +declaring a new stored variable, and the subclass will *not* need to +initialize `color` as a separate member variable. + +Because observing accessors add behavior to an inherited member +variable, a superclass's variable may not be overridden with a new +stored variable, even if no observing accessors are specified. In the +rare case where this is desired, the two-variable pattern shown +[above](%60Observing%20Accessors%60_) can be used. diff --git a/docs/StoredAndComputedVariables.rst b/docs/StoredAndComputedVariables.rst deleted file mode 100644 index 42bd2084979de..0000000000000 --- a/docs/StoredAndComputedVariables.rst +++ /dev/null @@ -1,287 +0,0 @@ -.. @raise litre.TestsAreMissing - -============================= -Stored and Computed Variables -============================= - -.. warning:: This document has not been updated since the initial design in - Swift 1.0. - - -Variables are declared using the ``var`` keyword. These declarations are valid -at the top level, within types, and within code bodies, and are respectively -known as *global variables,* *member variables,* and *local variables.* -Member variables are commonly referred to as *properties.* - -Every variable declaration can be classified as either *stored* or *computed.* -Member variables inherited from a superclass obey slightly different rules. - -.. contents:: :local: - - -Stored Variables -================ - -The simplest form of a variable declaration provides only a type:: - - var count : Int - -This form of ``var`` declares a *stored variable.* Stored variables cause -storage to be allocated in their containing context: - -- a new global symbol for a global variable -- a slot in an object for a member variable -- space on the stack for a local variable - -(Note that this storage may still be optimized away if determined unnecessary.) - -Stored variables must be initialized before use. As such, an initial value can -be provided at the declaration site. This is mandatory for global variables, -since it cannot be proven who accesses the variable first. :: - - var count : Int = 10 - -If the type of the variable can be inferred from the initial value expression, -it may be omitted in the declaration:: - - var count = 10 - -Variables formed during pattern matching are also considered stored -variables. :: - - switch optVal { - case .Some(var actualVal): - // do something - case .None: - // do something else - } - - -Computed Variables -================== - -A *computed variable* behaves syntactically like a variable, but does not -actually require storage. Instead, accesses to the variable go through -"accessors" known as the *getter* and the *setter.* Thus, a computed variable -is declared as a variable with a custom getter:: - - struct Rect { - // Stored member variables - var x, y, width, height : Int - - // A computed member variable - var maxX : Int { - get { - return x + width - } - set(newMax) { - x = newMax - width - } - } - - // myRect.maxX = 40 - -In this example, no storage is provided for ``maxX``. - -If the setter's argument is omitted, it is assumed to be named ``value``:: - - var maxY : Int { - get { - return y + height - } - set { - y = value - height - } - } - -Finally, if a computed variable has a getter but no setter, it becomes a -*read-only variable.* In this case the ``get`` label may be omitted. -Attempting to set a read-only variable is a compile-time error:: - - var area : Int { - return self.width * self.height - } - } - -Note that because this is a member variable, the implicit parameter ``self`` is -available for use within the accessors. - -It is illegal for a variable to have a setter but no getter. - - -Observing Accessors -=================== - -Occasionally it is useful to provide custom behavior when changing a variable's -value that goes beyond simply modifying the underlying storage. One way to do -this is to pair a stored variable with a computed variable:: - - var _backgroundColor : Color - var backgroundColor : Color { - get { - return _backgroundColor - } - set { - _backgroundColor = value - refresh() - } - } - -However, this contains a fair amount of boilerplate. For cases where a stored -property provides the correct storage semantics, you can add custom behavior -before or after the underlying assignment using "observing accessors" -``willSet`` and ``didSet``:: - - var backgroundColor : Color { - didSet { - refresh() - } - } - - var currentURL : URL { - willSet(newValue) { - if newValue != currentURL { - cancelCurrentRequest() - } - } - didSet { - sendNewRequest(currentURL) - } - } - -A stored property may have either observing accessor, or both. Like ``set``, -the argument for ``willSet`` may be omitted, in which case it is provided as -"value":: - - var accountName : String { - willSet { - assert(value != "root") - } - } - -Observing accessors provide the same behavior as the two-variable example, with -two important exceptions: - -- A variable with observing accessors is still a stored variable, which means - it must still be initialized before use. Initialization does not run the - code in the observing accessors. -- All assignments to the variable will trigger the observing accessors with - the following exceptions: assignments in the init and destructor function for - the enclosing type, and those from within the accessors themselves. In this - context, assignments directly store to the underlying storage. - -Computed properties may not have observing accessors. That is, a property may -have a custom getter or observing accessors, but not both. - - -Overriding Read-Only Variables -============================== - -If a member variable within a class is a read-only computed variable, it may -be overridden by subclasses. In this case, the subclass may choose to replace -that computed variable with a stored variable by declaring the stored variable -in the usual way:: - - class Base { - var color : Color { - return .Black - } - } - - class Colorful : Base { - var color : Color - } - - var object = Colorful(.Red) - object.color = .Blue - -The new stored variable may have observing accessors:: - - class MemoryColorful : Base { - var oldColors : Array = [] - - var color : Color { - willSet { - oldColors.append(color) - } - } - } - -A computed variable may also be overridden with another computed variable:: - - class MaybeColorful : Base { - var color : Color { - get { - if randomBooleanValue() { - return .Green - } else { - return super.color - } - } - set { - print("Sorry, we choose our own colors here.") - } - } - } - - -Overriding Read-Write Variables -=============================== - -If a member variable within a class as a read-write variable, it is not -generally possible to know if it is a computed variable or stored variable. -A subclass may override the superclass's variable with a new computed variable:: - - class ColorBase { - var color : Color { - didSet { - print("I've been painted \(color)!") - } - } - } - - class BrightlyColored : ColorBase { - var color : Color { - get { - return super.color - } - set(newColor) { - // Prefer whichever color is brighter. - if newColor.luminance > super.color.luminance { - super.color = newColor - } else { - // Keep the old color. - } - } - } - } - -In this case, because the superclass's ``didSet`` is part of the generated -setter, it is only called when the subclass actually invokes setter through -its superclass. On the ``else`` branch, the superclass's ``didSet`` is skipped. - -A subclass may also use observing accessors to add behavior to an inherited -member variable:: - - class TrackingColored : ColorBase { - var prevColor : Color? - - var color : Color { - willSet { - prevColor = color - } - } - } - -In this case, the ``willSet`` accessor in the subclass is called first, then -the setter for ``color`` in the superclass. Critically, this is *not* declaring -a new stored variable, and the subclass will *not* need to initialize ``color`` -as a separate member variable. - -Because observing accessors add behavior to an inherited member variable, a -superclass's variable may not be overridden with a new stored variable, even -if no observing accessors are specified. In the rare case where this is -desired, the two-variable pattern shown above__ can be used. - -__ `Observing Accessors`_ - diff --git a/docs/StringDesign.md b/docs/StringDesign.md new file mode 100644 index 0000000000000..b84742c5585c6 --- /dev/null +++ b/docs/StringDesign.md @@ -0,0 +1,1400 @@ + +.. role:: look1 .. role:: aside .. role:: emph + +Swift String Design +=================== + +> **This Document** +> +> - contains interactive HTML commentary that does not currently +> appear in printed output. Hover your mouse over elements with a +> dotted pink underline to view the hidden commentary. +> - represents the intended design of Swift strings, not their current +> implementation state. +> - is being delivered in installments. Content still to come is +> outlined in Coming Installments\_. + +> **warning** +> +> This document was used in planning Swift 1.0; it has not been kept + +> up to date and does not describe the current or planned behavior of +> Swift. + +Introduction +------------ + +Like all things Swift, our approach to strings begins with a deep +respect for the lessons learned from many languages and libraries, +especially Objective-C and Cocoa. + +### Goals + +`String` should: + +- honor industry standards such as Unicode +- when handling non-ASCII text, deliver “reasonably correct” results + to users thinking only in terms of ASCII +- when handling ASCII text, provide “expected behavior” to users + thinking only in terms of ASCII +- be hard to use incorrectly +- be easy to use correctly +- provide near-optimal efficiency for 99% of use cases +- provide a foundation upon which proper locale-sensitive operations + can be built + +### Non-Goals + +`String` need not: + +- have behavior appropriate to all locales and contexts +- be an appropriate type (or base type) for all text storage + applications + +Overview By Example +------------------- + +In this section, we'll walk through some basic examples of Swift string +usage while discovering its essential properties. + +### `String` is a [First-Class Type](http://en.wikipedia.org/wiki/First-class_citizen) + +Unlike, say, C's `char*`, the meaning of a swift string is always +unambiguous. + +### Strings are **Efficient** + +The implementation of `String` takes advantage of state-of-the-art +optimizations, including: + +- Storing short strings without heap allocation +- Sharing allocated buffers among copies and slices +- In-place modification of uniquely-owned buffers + +As a result, copying\_ and [slicing](sliceable_) strings, in particular, +can be viewed by most programmers as being “almost free.” + +### Strings are **Mutable** + +> **Why Mention It?** +> +> The ability to change a string's value might not be worth noting +> except that *some languages make all strings immutable*, as a way of +> working around problems that Swift has defined away—by making strings +> pure values (see below). + +> (swift) extension String { +> +> : +> +> func addEcho() { +> +> : self += self +> +> } +> +> > } +> +> (swift) :look1: class="repl">s.addEcho()s is modified in place (swift) s class="repl">// s: String = "YoYo" + +### Strings are **Value Types** + +Distinct string variables have independent values: when you pass someone +a string they get a copy of the value, and when someone passes you a +string *you own it*. Nobody can change a string value “behind your +back.” + +> (swift) class Cave { +> +> : // Utter something in the cave func say(msg: String) -> String +> { :look1: class="repl">msg.addEcho()Modifying a parameter is safe because the callee sees a copy of the argument +> self.lastSound = msg :look1: class="repl">return self.lastSoundReturning a stored value is safe because the caller sees a copy of the value +> } +> +> var lastSound: String // a Cave remembers the last sound made +> +> > } +> +> (swift) var c = Cave() class="repl">// c: Cave = <Cave instance> (swift) s = +> "Hey" (swift) var t = :look1: class="repl">c.say(s)this call can't change s… class="repl">// t: String = "HeyHey" (swift) s class="repl">// s: String = class="look">"Hey"…and it doesn't. (swift) :look1: class="repl">t.addEcho()this call can't change c.lastSound… +> (swift) \[s, c.lastSound, t\] class="repl">// r0: String\[\] = \["Hey", class="look">"HeyHey"…and it doesn't. class="repl">, "HeyHeyHeyHey"\] + +### Strings are **Unicode-Aware** + +> **Deviations from Unicode** +> +> Any deviation from what Unicode specifies requires careful +> justification. So far, we have found two possible points of deviation +> for Swift `String`: +> +> 1. The [Unicode Text Segmentation +> Specification](http://www.unicode.org/reports/tr29) says, “[do not +> break between CR and +> LF](http://www.unicode.org/glossary/#grapheme_cluster).” However, +> breaking extended grapheme clusters between CR and LF may +> necessary if we wish `String` to “behave normally” for users of +> pure ASCII. This point is still open for discussion. +> +> \_\_ +> +> 2. The [Unicode Text Segmentation +> Specification](http://www.unicode.org/reports/tr29) says, “[do not +> break between regional indicator +> symbols](http://useless-factor.blogspot.com/2007/08/unicode-implementers-guide-part-4.html).” +> However, it also says “(Sequences of more than two RI characters +> should be separated by other characters, such as U+200B ZWSP).” +> Although the parenthesized note probably has less official weight +> than the other admonition, breaking pairs of RI characters seems +> like the right thing for us to do given that Cocoa already forms +> strings with several adjacent pairs of RI characters, and the +> Unicode spec *can* be read as outlawing such strings anyway. +> +> \_\_ +> +Swift applies Unicode algorithms wherever possible. For example, +distinct sequences of code points are treated as equal if they represent +the same character: [^1] + +> (swift) var n1 = ":look1: class="repl">\\\\u006E\\\\u0303Multiple code points, but only one Character" +> // n1 : String = **"ñ"** (swift) var n2 = +> "\\u00F1" // n2 : String = **"ñ"** (swift) +> n1 == n2 // r0 : Bool = **true** + +Note that individual code points are still observable by explicit +request: + +> (swift) n1.codePoints == n2.codePoints class="repl">// r0 : Bool = **false** + +### Strings are **Locale-Agnostic** + +Strings neither carry their own locale information, nor provide +behaviors that depend on a global locale setting. Thus, for any pair of +strings `s1` and `s2`, “`s1 == s2`” yields the same result regardless of +system state. Strings *do* provide a suitable foundation on which to +build locale-aware interfaces.[^2] + +### Strings are **Containers** + +> **String Indices** +> +> `String` implements the `Container` protocol, but **cannot be indexed +> by integers**. Instead, `String.IndexType` is a library type +> conforming to the `BidirectionalIndex` protocol. +> +> This might seem surprising at first, but code that indexes strings +> with arbitrary integers is seldom Unicode-correct in the first place, +> and Swift provides alternative interfaces that encourage +> Unicode-correct code. For example, instead of `s[0] == 'S'` you'd +> write `s.startsWith("S")`. + +### Strings are Composed of `Character`s + +`Character`, the element type of `String`, represents a **grapheme +cluster**, as specified by a default or tailored Unicode segmentation +algorithm. This term is [precisely +defined](http://www.unicode.org/reports/tr29/#Default_Grapheme_Cluster_Table) +by the Unicode specification, but it roughly means [what the user thinks +of when she hears +“character”](http://www.unicode.org/glossary/#code_unit). For example, +the pair of code points “LATIN SMALL LETTER N, COMBINING TILDE” forms a +single grapheme cluster, “ñ”. + +Access to lower-level elements is still possible by explicit request: + +Strings Support Flexible Segmentation +------------------------------------- + +The `Character`s enumerated when simply looping over elements of a Swift +string are [extended grapheme clusters](length_) as determined by +Unicode's Default Grapheme Cluster Boundary +Specification\_\_. [^3] + +This segmentation offers naïve users of English, Chinese, French, and +probably a few other languages what we think of as the “expected +results.” However, not every +[script](http://www.unicode.org/glossary/#script) can be segmented +uniformly for all purposes. For example, searching and collation require +different segmentations in order to handle Indic scripts correctly. To +that end, strings support properties for more-specific segmentations: + +> **note** +> +> The following example needs a more interesting string in +> +> : order to demonstrate anything interesting. Hopefully Aki has some +> advice for us. +> +Also, each such segmentation provides a unique `IndexType`, allowing a +string to be indexed directly with different indexing schemes: + + |swift| var i = s.searchCharacters.startIndex + `// r2 : UInt8 = UInt8(83)` + +### Strings are **Sliceable** + +### Strings are **Encoded as UTF-8** + +> **Encoding Conversion** +> +> Conversion to and from other encodings is out-of-scope for `String` +> itself, but could be provided, e.g., by an `Encoding` module. + +Coming Installments +------------------- + +- Reference Manual +- Rationales +- Cocoa Bridging Strategy +- Comparisons with NSString + - High Level + - Member-by-member + +Reference Manual +---------------- + +- s.bytes +- s.indices +- s\[i\] +- s\[start...end\] +- s == t, s != t +- s < t, s > t, s <= t, s >= t +- s.hash() +- s.startsWith(), s.endsWith() +- s + t, s += t, s.append(t) +- s.split(), s.split(n), s.split(sep, n) +- s.strip(), s.stripStart(), s.stripEnd() +- s.commonPrefix(t), s.mismatch(t) +- s.toUpper(), s.toLower() +- s.trim(predicate) +- s.replace(old, new, count) +- s.join(sequenceOfStrings) + +Cocoa Bridging Strategy +----------------------- + +Rationales +---------- + +### Why a Built-In String Type? + +> **DaveZ Sez** +> +> In the "why a built-in string type" section, I think the main +> narrative is that two string types is bad, but that we have two string +> types in Objective-C for historically good reasons. To get one string +> type, we need to merge the high-level features of Objective-C with the +> performance of C, all while not having the respective bad the bad +> semantics of either (reference semantics and "anarchy" +> memory-management respectively). Furthermore, I'd write "value +> semantics" in place of "C++ semantics". I know that is what you meant, +> but we need to tread carefully in the final document. + +`NSString` and `NSMutableString`—the string types provided by Cocoa—are +full-featured classes with high-level functionality for writing +fully-localized applications. They have served Apple programmers well; +so, why does Swift have its own string type? + +- ObjCMessageSend +- Error Prone Mutability Reference semantics don't line up with how + people think about strings +- 2 is too many string types. two APIs duplication of effort + documentation Complexity adds decisions for users etc. +- ObjC needed to innovate because C strings suck O(N) length no + localization no memory management no specified encoding +- C strings had to stay around for performance reasons and + interoperability + +Want performance of C, sane semantics of C++ strings, and high-level +goodness of ObjC. + +> The design of `NSString` is *very* different from the string designs +> of most modern programming languages, which all tend to be very +> similar to one another. Although existing `NSString` users are a +> critical constituency today, current trends indicate that most of our +> *future* target audience will not be `NSString` users. Absent +> compelling justification, it's important to make the Swift programming +> environment as familiar as possible for them. + +### How Would You Design It? + +> **DaveZ Sez** +> +> In the "how would you design it" section, the main narrative is +> twofold: how does it "feel" and how efficient is it? The former is +> about feeling built in, which we can easily argue that both C strings +> or Cocoa strings fail at for their respective semantic (and often +> memory management related) reasons. Additionally, the "feel" should be +> modern, which is where the Cocoa framework and the Unicode standard +> body do better than C. Nevertheless, we can still do better than +> Objective-C and your strong work at helping people reason about +> grapheme clusters instead of code points (or worse, units) is +> wonderful and it feels right to developers. The second part of the +> narrative is about being efficient, which is where arguing for UTF8 is +> the non-obvious but "right" answer for the reasons we have discussed. + +- It'd be an independent *value* so you don't have to micromanage + sharing and mutation +- It'd be UTF-8 because: + - UTF-8 has been the clear winner\_\_ among Unicode encodings + since at least 2008; Swift should interoperate smoothly and + efficiently with the rest of the world's systems + + \_\_ + + - UTF-8 is a fairly efficient storage format, especially for ASCII + but also for the most common non-ASCII code points. + - This\_\_ posting elaborates on some other nice qualities of + UTF-8: + + 1. All ASCII files are already UTF-8 files + 2. ASCII bytes always represent themselves in UTF-8 files. They + never appear as part of other UTF-8 sequences + 3. ASCII code points are always represented as themselves in + UTF-8 files. They cannot be hidden inside multibyte UTF-8 + sequences + 4. UTF-8 is self-synchronizing + 5. CodePoint substring search is just byte string search + 6. Most programs that handle 8-bit files safely can handle + UTF-8 safely + 7. UTF-8 sequences sort in code point order. + 8. UTF-8 has no “byte order.” + + \_\_ + + +- It would be efficient, taking advantage of state-of-the-art + optimizations, including: + - Storing short strings without heap allocation + - Sharing allocated buffers among copies and slices + - In-place modification of uniquely-owned buffers + +Comparisons with `NSString` +--------------------------- + +### High-Level Comparison with `NSString` + +> **DaveZ Sez** +> +> I think the main message of the API breadth subsection is that URLs, +> paths, etc would be modeled as formal types in Swift (i.e. not as +> extensions on String). Second, I'd speculate less on what Foundation +> could do (like extending String) and instead focus on the fact that +> NSString still exists as an escape hatch for those that feel that they +> need or want it. Furthermore, I'd move up the "element access" +> discussion above the "escape hatch" discussion (which should be last +> in the comparison with NSString discussion). + +#### API Breadth + +The `NSString` interface clearly shows the effects of 20 years of +evolution through accretion. It is broad, with functionality addressing +encodings, paths, URLs, localization, and more. By contrast, the +interface to Swift's `String` is much narrower. + +Of course, there's a reason for every `NSString` method, and the full +breadth of `NSString` functionality must remain accessible to the +Cocoa/Swift programmer. Fortunately, there are many ways to address this +need. For example: + +- The `Foundation` module can extend `String` with the methods of + `NSString`. The extent to which we provide an identical-feeling + interface and/or correct any `NSString` misfeatures is still TBD and + wide open for discussion. +- We can create a new modular interface in pure Swift, including a + `Locale` module that addresses localized string operations, an + `Encoding` module that addresses character encoding schemes, a + `Regex` module that provides regular expression functionality, etc. + Again, the specifics are TBD. +- When all else fails, users can convert their Swift `String`s to + `NSString`s when they want to access `NSString`-specific + functionality: + +For Swift version 1.0, we err on the side of keeping the string +interface small, coherent, and sufficient for implementing higher-level +functionality. + +#### Element Access + +`NSString` exposes UTF-16 code units\_\_ as +the primary element on which indexing, slicing, and iteration operate. +Swift's UTF-8 code units are only available as a secondary interface. + +`NSString` is indexable and sliceable using `Int`s, and so exposes a +`length` attribute. Swift's `String` is indexable and sliceable using an +abstract `BidirectionalIndex` type, and does not +expose its length\_\_. + +#### Sub-Strings + +Creating substrings in Swift is very fast. Therefore, Cocoa APIs that +operate on a substring given as an `NSRange` are replaced with Swift +APIs that just operate on `String`s. One can use range-based +subscripting to achieve the same effect. For example: +`[str doFoo:arg withRange:subrange]` becomes `str[subrange].doFoo(arg)`. + +### `NSString` Member-by-Member Comparison + +Notes + +: - The following are from public headers from public frameworks, + which are AppKit and Foundation (verified). + - Deprecated Cocoa APIs are not considered + - A status of “*Remove*” below indicates a feature whose removal + is anticipated. Rationale is provided for these cases. + +#### Indexing + +------------------------------------------------------------------------ + +> **Why doesn't `String` support `.length`?** +> +> In Swift, by convention, `x.length` is used to represent the number of +> elements in a container, and since `String` is a container of abstract +> `Character`\_s, `length` would have to count those. +> +> This meaning of `length` is unimplementable in O(1). It can be cached, +> although not in the memory block where the characters are stored, +> since we want a `String` to share storage with its slices. Since the +> body of the `String` must already store the `String`'s *byte length*, +> caching the `length` would increase the footprint of the top-level +> String object. Finally, even if `length` were provided, doing things +> with `String` that depend on a specific numeric `length` is +> error-prone. + +Cocoa + +: + +Swift + +: *not directly provided*, but similar functionality is available: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: typealias IndexType = ... func **indices**() -> + > Range<IndexType> **subscript**(i: IndexType) -> Character + + > **Usage** + +#### Slicing + +Cocoa + +: + +Swift + +: typealias IndexType = ... **subscript**(r: Range<IndexType>) + > -> Character + +#### Indexing + +Cocoa + +: + +Swift + +: **subscript**(range : Range<IndexType>) -> String + + > **Example** + > + > Note + > + > : Swift may need additional interfaces to support `index...` and + > `...index` notations. This part of the `Container` protocol + > design isn't worked out yet. + > +#### Comparison + +Cocoa + +: + +Swift + +: + +`NSString` comparison is “literal” by default. As the documentation says +of `isEqualToString`, + +> “Ö” represented as the composed character sequence “O” and umlaut +> would not compare equal to “Ö” represented as one Unicode character. + +By contrast, Swift string's primary comparison interface uses Unicode's +default [collation](http://www.unicode.org/reports/tr10/) algorithm, and +is thus always “Unicode-correct.” Unlike comparisons that depend on +locale, it is also stable across changes in system state. However, *just +like* `NSString`'s `isEqualToString` and `compare` methods, it should +not be expected to yield ideal (or even “proper”) results in all +contexts. + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: *various compositions of primitive operations* / TBD\_ + +- As noted above\_\_, instead of passing sub-range arguments, we + expect Swift users to compose slicing\_ with + whole-string operations. + + \_\_ range\_ + +- Other details of these interfaces are distinguished by an + `NSStringCompareOptions` mask, of which `caseInsensitiveCompare:` is + essentially a special case: + + `NSCaseInsensitiveSearch` + + : Whether a direct interface is needed at all in Swift, and if so, + its form, are TBD\_. However, we should consider following the + lead of Python 3, wherein case conversion also normalizes letterforms\_\_. Then one can + combine `String.toLower()` with default comparison to get a + case-insensitive comparison: + + { $0.toLower() == $1.toLower() } + + \_\_ + + `NSLiteralSearch` + + : Though it is the default for `NSString`, this option is + essentially only useful as a performance optimization when the + string content is known to meet certain restrictions (i.e. is + known to be pure ASCII). When such optimization is absolutely + necessary, Swift standard library algorithms can be used + directly on the `String`'s UTF8 code units. However, Swift will + also perform these optimizations automatically (at the cost of a + single test/branch) in many cases, because each `String` stores + a bit indicating whether its content is known to be ASCII. + + `NSBackwardsSearch` + + : It's unclear from the docs how this option interacts with other + `NSString` options, if at all, but basic cases can be handled in + Swift by `s1.endsWith(s2)`. + + `NSAnchoredSearch` + + : Not applicable to whole-string comparisons + + `NSNumericSearch` + + : While it's legitimate to defer this functionality to Cocoa, it's + (probably—see <rdar://problem/14724804>) + locale-independent and easy enough to implement in Swift. TBD\_ + + `NSDiacriticInsensitiveSearch` + + : Ditto; TBD\_ + + `NSWidthInsensitiveSearch` + + : Ditto; TBD\_ + + `NSForcedOrderingSearch` + + : Ditto; TBD\_. Also see <rdar://problem/14724888> + + `NSRegularExpressionSearch` + + : We can defer this functionality to Cocoa, or dispatch directly + to ICU as an optimization. It's unlikely that we'll be building + Swift its own regexp engine for 1.0. + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: As these all depend on locale, they are TBD\_ + +#### Searching + +> **Rationale** +> +> Modern languages (Java, C\#, Python, Ruby…) have standardized on +> variants of `startsWith`/`endsWith`. There's no reason Swift should +> deviate from de-facto industry standards here. + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: **note** + > + > Most other languages provide something like + > + > : `s1.indexOf(s2)`, which returns only the starting index of the + > first match. This is far less useful than the range of the + > match, and is always available via `s1.find(s2).bounds.0` + > +------------------------------------------------------------------------ + +Cocoa + +: + +> **Naming** +> +> The Swift function is just an algorithm that comes from conformance to +> the `Container` protocol, which explains why it doesn't have a +> `String`-specific name. + +Swift + +: **Usage Example** + > + > The `NSString` semantics can be achieved as follows: + +------------------------------------------------------------------------ + +Cocoa + +: These functions + +Swift + +: *various compositions of primitive operations* / TBD\_ + +#### Building + +Cocoa + +: + +> **`append`** +> +> the `append` method is a consequence of `String`'s conformance to +> `OutputStream`. See the *Swift formatting proposal* for details. + +Swift + +: + +#### Dynamic Formatting + +Cocoa + +: + +Swift + +: *Not directly provided*—see the *Swift formatting proposal* + +#### Extracting Numeric Values + +Cocoa + +: + +Swift + +: Not in `String`—It is up to other types to provide their conversions + to and from String. See also this rationale\_\_ + + \_\_ extending\_ + +#### Splitting + +Cocoa + +: + +Swift + +: The semantics of these functions were taken from Python, which seems + to be a fairly good representative of what modern languages are + currently doing. The first overload splits on all whitespace + characters; the second only on specific characters. The universe of + possible splitting functions is quite broad, so the particulars of + this interface are **wide open for discussion**. In Swift right now, + these methods (on `CodePoints`) are implemented in terms of a + generic algorithm: + +#### Splitting + +Cocoa + +: + +Swift + +: + +#### Upper/Lowercase + +Cocoa + +: + +> **Naming** +> +> Other languages have overwhelmingly settled on `upper()` or +> `toUpper()` for this functionality + +Swift + +: + +#### Capitalization + +Cocoa + +: + +Swift + +: **TBD** + +> **note** +> +> `NSString` capitalizes the first letter of each substring +> +> : separated by spaces, tabs, or line terminators, which is in no +> sense “Unicode-correct.” In most other languages that support a +> `capitalize` method, it operates only on the first character of +> the string, and capitalization-by-word is named something like +> “`title`.” If Swift `String` supports capitalization by word, it +> should be Unicode-correct, but how we sort this particular area +> out is still **TBD**. +> +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: **Usage Example** + > + > The `NSString` semantics can be achieved as follows: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +- (BOOL)**canBeConvertedToEncoding:**(NSStringEncoding)encoding; + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +#### Constructors + +Cocoa + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +------------------------------------------------------------------------ + +Cocoa + +: + (instancetype)string; + +------------------------------------------------------------------------ + +Cocoa + +: + (instancetype)**stringWithString:**(NSString \*)string; + +Not available (too error prone) + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +Swift + +: + +------------------------------------------------------------------------ + +Cocoa + +: + +#### Linguistic Analysis + +Cocoa + +: + +Swift + +: + +### Unavailable on Swift Strings + +#### URL Handling + +See: class File + +#### Path Handling + +#### Property Lists + +Property lists are a feature of Cocoa. + +#### Deprecated APIs + +Already deprecated in Cocoa. + +------------------------------------------------------------------------ + +### Why YAGNI + +- Retroactive Modeling +- Derivation +- ... + +[^1]: Technically, `==` checks for [Unicode canonical + equivalence](http://www.unicode.org/glossary/#extended_grapheme_cluster) + + \_\_ + +[^2]: We have some specific ideas for locale-sensitive interfaces, but + details are still TBD and wide open for discussion. + +[^3]: The type currently called `Char` in Swift represents a Unicode + code point. This document refers to it as `CodePoint`, in + anticipation of renaming. diff --git a/docs/StringDesign.rst b/docs/StringDesign.rst deleted file mode 100644 index 52fcfca9d8b6b..0000000000000 --- a/docs/StringDesign.rst +++ /dev/null @@ -1,1753 +0,0 @@ -:orphan: - -.. @raise litre.TestsAreMissing - -.. raw:: html - - - -.. role:: repl -.. default-role:: repl - -.. |swift| replace:: (swift) - -.. role:: look -.. role:: look1 -.. role:: aside -.. role:: emph - -=================== -Swift String Design -=================== - -.. Admonition:: This Document - :class: note - - * contains interactive HTML commentary that does not - currently appear in printed output. Hover your mouse over - elements with a dotted pink underline to view the hidden - commentary. - - * represents the intended design of Swift strings, not their - current implementation state. - - * is being delivered in installments. Content still to come is - outlined in `Coming Installments`_. - -.. warning:: This document was used in planning Swift 1.0; it has not been kept - up to date and does not describe the current or planned behavior of Swift. - -.. contents:: - :depth: 3 - -Introduction -============ - -Like all things Swift, our approach to strings begins with a deep -respect for the lessons learned from many languages and libraries, -especially Objective-C and Cocoa. - -Goals ------ - -``String`` should: - -* honor industry standards such as Unicode -* when handling non-ASCII text, deliver “reasonably correct” - results to users thinking only in terms of ASCII -* when handling ASCII text, provide “expected behavior” to users - thinking only in terms of ASCII -* be hard to use incorrectly -* be easy to use correctly -* provide near-optimal efficiency for 99% of use cases -* provide a foundation upon which proper locale-sensitive operations - can be built - -Non-Goals ---------- - -``String`` need not: - -* have behavior appropriate to all locales and contexts -* be an appropriate type (or base type) for all text storage - applications - -Overview By Example -=================== - -In this section, we'll walk through some basic examples of Swift -string usage while discovering its essential properties. - -``String`` is a `First-Class Type`__ ------------------------------------- - -__ http://en.wikipedia.org/wiki/First-class_citizen - -.. parsed-literal:: - - |swift| var s = "Yo" - `// s:` :emph:`String` `= "Yo"` - -Unlike, say, C's ``char*``, the meaning of a swift string is always -unambiguous. - -Strings are **Efficient** -------------------------- - -The implementation of ``String`` takes advantage of state-of-the-art -optimizations, including: - -- Storing short strings without heap allocation -- Sharing allocated buffers among copies and slices -- In-place modification of uniquely-owned buffers - -As a result, copying_ and slicing__ strings, in particular, can be -viewed by most programmers as being “almost free.” - -__ sliceable_ - -Strings are **Mutable** ------------------------ - -.. sidebar:: Why Mention It? - - The ability to change a string's value might not be worth noting - except that *some languages make all strings immutable*, as a way - of working around problems that Swift has defined away—by making - strings pure values (see below). - -.. parsed-literal:: - |swift| extension String { - func addEcho() { - self += self - } - } - |swift| :look1:`s.addEcho()`\ :aside:`s is modified in place` - |swift| s - `// s: String =` :emph:`"YoYo"` - -.. _copying: - -Strings are **Value Types** ---------------------------- - -Distinct string variables have independent values: when you pass -someone a string they get a copy of the value, and when someone -passes you a string *you own it*. Nobody can change a string value -“behind your back.” - -.. parsed-literal:: - |swift| class Cave { - // Utter something in the cave - func say(msg: String) -> String { - :look1:`msg.addEcho()`\ :aside:`Modifying a parameter is safe because the callee sees a copy of the argument` - self.lastSound = msg - :look1:`return self.lastSound`\ :aside:`Returning a stored value is safe because the caller sees a copy of the value` - } - - var lastSound: String // a Cave remembers the last sound made - } - |swift| var c = Cave() - `// c: Cave = ` - |swift| s = "Hey" - |swift| var t = :look1:`c.say(s)`\ :aside:`this call can't change s…` - `// t: String = "HeyHey"` - |swift| s - `// s: String =` :look:`"Hey"`\ :aside:`…and it doesn't.` - |swift| :look1:`t.addEcho()`\ :aside:`this call can't change c.lastSound…` - |swift| [s, c.lastSound, t] - `// r0: String[] = ["Hey",` :look:`"HeyHey"`\ :aside:`…and it doesn't.`\ `, "HeyHeyHeyHey"]` - -Strings are **Unicode-Aware** ------------------------------ - -.. sidebar:: Deviations from Unicode - - - Any deviation from what Unicode - specifies requires careful justification. So far, we have found two - possible points of deviation for Swift ``String``: - - 1. The `Unicode Text Segmentation Specification`_ says, “`do not - break between CR and LF`__.” However, breaking extended - grapheme clusters between CR and LF may necessary if we wish - ``String`` to “behave normally” for users of pure ASCII. This - point is still open for discussion. - - __ http://www.unicode.org/reports/tr29/#GB2 - - 2. The `Unicode Text Segmentation Specification`_ says, - “`do not break between regional indicator symbols`__.” However, it also - says “(Sequences of more than two RI characters should be separated - by other characters, such as U+200B ZWSP).” Although the - parenthesized note probably has less official weight than the other - admonition, breaking pairs of RI characters seems like the right - thing for us to do given that Cocoa already forms strings with - several adjacent pairs of RI characters, and the Unicode spec *can* - be read as outlawing such strings anyway. - - __ http://www.unicode.org/reports/tr29/#GB8 - -.. _Unicode Text Segmentation Specification: http://www.unicode.org/reports/tr29 - -Swift applies Unicode algorithms wherever possible. For example, -distinct sequences of code points are treated as equal if they -represent the same character: [#canonical]_ - -.. parsed-literal:: - |swift| var n1 = ":look1:`\\u006E\\u0303`\ :aside:`Multiple code points, but only one Character`" - `// n1 : String =` **"ñ"** - |swift| var n2 = "\\u00F1" - `// n2 : String =` **"ñ"** - |swift| n1 == n2 - `// r0 : Bool =` **true** - -Note that individual code points are still observable by explicit request: - -.. parsed-literal:: - |swift| n1.codePoints == n2.codePoints - `// r0 : Bool =` **false** - -.. _locale-agnostic: - -Strings are **Locale-Agnostic** -------------------------------- - -Strings neither carry their own locale information, nor provide -behaviors that depend on a global locale setting. Thus, for any pair -of strings ``s1`` and ``s2``, “``s1 == s2``” yields the same result -regardless of system state. Strings *do* provide a suitable -foundation on which to build locale-aware interfaces.\ [#locales]_ - -Strings are **Containers** --------------------------- - -.. sidebar:: String Indices - - ``String`` implements the ``Container`` protocol, but - **cannot be indexed by integers**. Instead, - ``String.IndexType`` is a library type conforming to the - ``BidirectionalIndex`` protocol. - - This might seem surprising at first, but code that indexes - strings with arbitrary integers is seldom Unicode-correct in - the first place, and Swift provides alternative interfaces - that encourage Unicode-correct code. For example, instead - of ``s[0] == 'S'`` you'd write ``s.startsWith("S")``. - -.. parsed-literal:: - |swift| var s = "Strings are awesome" - `// s : String = "Strings are awesome"` - |swift| var r = s.find("awe") - `// r : Range = <"…are a̲w̲e̲some">` - |swift| s[r.start] - `// r0 : Character =` :look:`Character("a")`\ :aside:`String elements have type Character (see below)` - -.. |Character| replace:: ``Character`` -.. _Character: - -Strings are Composed of ``Character``\ s ----------------------------------------- - -``Character``, the element type of ``String``, represents a **grapheme -cluster**, as specified by a default or tailored Unicode segmentation -algorithm. This term is `precisely defined`__ by the Unicode -specification, but it roughly means `what the user thinks of when she -hears “character”`__. For example, the pair of code points “LATIN -SMALL LETTER N, COMBINING TILDE” forms a single grapheme cluster, “ñ”. - -__ http://www.unicode.org/glossary/#grapheme_cluster -__ http://useless-factor.blogspot.com/2007/08/unicode-implementers-guide-part-4.html - -Access to lower-level elements is still possible by explicit request: - -.. parsed-literal:: - |swift| s.codePoints[s.codePoints.start] - `// r1 : CodePoint = CodePoint(83) /* S */` - |swift| s.bytes[s.bytes.start] - `// r2 : UInt8 = UInt8(83)` - -Strings Support Flexible Segmentation -===================================== - -The ``Character``\ s enumerated when simply looping over elements of a -Swift string are `extended grapheme clusters`__ as determined by -Unicode's `Default Grapheme Cluster Boundary -Specification`__. [#char]_ - -__ http://www.unicode.org/glossary/#extended_grapheme_cluster -__ http://www.unicode.org/reports/tr29/#Default_Grapheme_Cluster_Table - -This segmentation offers naïve users of English, Chinese, French, and -probably a few other languages what we think of as the “expected -results.” However, not every script_ can be segmented uniformly for -all purposes. For example, searching and collation require different -segmentations in order to handle Indic scripts correctly. To that -end, strings support properties for more-specific segmentations: - -.. Note:: The following example needs a more interesting string in - order to demonstrate anything interesting. Hopefully Aki - has some advice for us. - -.. parsed-literal:: - |swift| for c in s { print("Extended Grapheme Cluster: \(c)") } - `Extended Grapheme Cluster: f` - `Extended Grapheme Cluster: o` - `Extended Grapheme Cluster: o` - |swift| for c in s.collationCharacters { - print("Collation Grapheme Cluster: \(c)") - } - `Collation Grapheme Cluster: f` - `Collation Grapheme Cluster: o` - `Collation Grapheme Cluster: o` - |swift| for c in s.searchCharacters { - print("Search Grapheme Cluster: \(c)") - } - `Search Grapheme Cluster: f` - `Search Grapheme Cluster: o` - `Search Grapheme Cluster: o` - -Also, each such segmentation provides a unique ``IndexType``, allowing -a string to be indexed directly with different indexing schemes:: - - |swift| var i = s.searchCharacters.startIndex - `// r2 : UInt8 = UInt8(83)` - -.. _script: http://www.unicode.org/glossary/#script - -.. _sliceable: - -Strings are **Sliceable** -------------------------- - -.. parsed-literal:: - |swift| s[r.start...r.end] - `// r2 : String = "awe"` - |swift| s[\ :look1:`r.start...`\ ]\ :aside:`postfix slice operator means “through the end”` - `// r3 : String = "awesome"` - |swift| s[\ :look1:`...r.start`\ ]\ :aside:`prefix slice operator means “from the beginning”` - `// r4 : String = "Strings are "` - |swift| :look1:`s[r]`\ :aside:`indexing with a range is the same as slicing` - `// r5 : String = "awe"` - |swift| s[r] = "hand" - |swift| s - `// s : String = "Strings are` :look:`handsome`\ :aside:`slice replacement can resize the string`\ `"` - -.. _extending: - -Strings are **Encoded as UTF-8** --------------------------------- - -.. sidebar:: Encoding Conversion - - Conversion to and from other encodings is out-of-scope for - ``String`` itself, but could be provided, e.g., by an ``Encoding`` - module. - -.. parsed-literal:: - |swift| for x in "bump"\ **.bytes** { - print(x) - } - 98 - 117 - 109 - 112 - -Coming Installments -=================== - -* Reference Manual - -* Rationales - -* Cocoa Bridging Strategy - -* Comparisons with NSString - - - High Level - - Member-by-member - -Reference Manual -================ - - -* s.bytes -* s.indices -* s[i] -* s[start...end] -* s == t, s != t -* s < t, s > t, s <= t, s >= t -* s.hash() -* s.startsWith(), s.endsWith() -* s + t, s += t, s.append(t) -* s.split(), s.split(n), s.split(sep, n) -* s.strip(), s.stripStart(), s.stripEnd() -* s.commonPrefix(t), s.mismatch(t) -* s.toUpper(), s.toLower() -* s.trim(predicate) -* s.replace(old, new, count) -* s.join(sequenceOfStrings) - -.. Stuff from Python that we don't need to do - - * s.capitalize() - * s.find(), s.rfind() - * Stuff for monospace - * s * 20 - * s.center() - * s.count() [no arguments] - * s.expandTabs(tabsize) - * s.leftJustify(width, fillchar) - * s.rightJustify(width, fillchar) - * s.count() - * s.isAlphanumeric() - * s.isAlphabetic() - * s.isNumeric() - * s.isDecimal() - * s.isDigit()? - * s.isLower() - * s.isUpper() - * s.isSpace() - * s.isTitle() - -Cocoa Bridging Strategy -======================= -.. - - -Rationales -========== - -Why a Built-In String Type? ---------------------------- - -.. Admonition:: DaveZ Sez - - In the "why a built-in string type" section, I think the main - narrative is that two string types is bad, but that we have two - string types in Objective-C for historically good reasons. To get - one string type, we need to merge the high-level features of - Objective-C with the performance of C, all while not having the - respective bad the bad semantics of either (reference semantics and - "anarchy" memory-management respectively). Furthermore, I'd write - "value semantics" in place of "C++ semantics". I know that is what - you meant, but we need to tread carefully in the final document. - -``NSString`` and ``NSMutableString``\ —the string types provided by -Cocoa—are full-featured classes with high-level functionality for -writing fully-localized applications. They have served Apple -programmers well; so, why does Swift have its own string type? - -* ObjCMessageSend - -* Error Prone Mutability - Reference semantics don't line up with how people think about strings - -* 2 is too many string types. - two APIs - duplication of effort - documentation - Complexity adds decisions for users - etc. - -* ObjC needed to innovate because C strings suck - O(N) length - no localization - no memory management - no specified encoding - -* C strings had to stay around for performance reasons and - interoperability - -Want performance of C, sane semantics of C++ strings, and high-level -goodness of ObjC. - - The design of ``NSString`` is *very* different from the string - designs of most modern programming languages, which all tend to be - very similar to one another. Although existing ``NSString`` users - are a critical constituency today, current trends indicate that - most of our *future* target audience will not be ``NSString`` - users. Absent compelling justification, it's important to make the - Swift programming environment as familiar as possible for them. - - -How Would You Design It? ------------------------- - -.. Admonition:: DaveZ Sez - - In the "how would you design it" section, the main narrative is - twofold: how does it "feel" and how efficient is it? The former is - about feeling built in, which we can easily argue that both C - strings or Cocoa strings fail at for their respective semantic (and - often memory management related) reasons. Additionally, the "feel" - should be modern, which is where the Cocoa framework and the - Unicode standard body do better than C. Nevertheless, we can still - do better than Objective-C and your strong work at helping people - reason about grapheme clusters instead of code points (or worse, - units) is wonderful and it feels right to developers. The second - part of the narrative is about being efficient, which is where - arguing for UTF8 is the non-obvious but "right" answer for the - reasons we have discussed. - -* It'd be an independent *value* so you don't have to micromanage - sharing and mutation - -* It'd be UTF-8 because: - - - UTF-8 has been the clear winner__ among Unicode encodings since at - least 2008; Swift should interoperate smoothly and efficiently - with the rest of the world's systems - - __ http://www.artima.com/weblogs/viewpost.jsp?thread=230157 - - - UTF-8 is a fairly efficient storage format, especially for ASCII - but also for the most common non-ASCII code points. - - - This__ posting elaborates on some other nice qualities of UTF-8: - - 1. All ASCII files are already UTF-8 files - 2. ASCII bytes always represent themselves in UTF-8 files. They - never appear as part of other UTF-8 sequences - 3. ASCII code points are always represented as themselves in UTF-8 - files. They cannot be hidden inside multibyte UTF-8 - sequences - 4. UTF-8 is self-synchronizing - 5. CodePoint substring search is just byte string search - 6. Most programs that handle 8-bit files safely can handle UTF-8 safely - 7. UTF-8 sequences sort in code point order. - 8. UTF-8 has no “byte order.” - - __ http://research.swtch.com/2010/03/utf-8-bits-bytes-and-benefits.html - -* It would be efficient, taking advantage of state-of-the-art - optimizations, including: - - - Storing short strings without heap allocation - - Sharing allocated buffers among copies and slices - - In-place modification of uniquely-owned buffers - - -Comparisons with ``NSString`` -============================= - -High-Level Comparison with ``NSString`` ---------------------------------------- - -.. Admonition:: DaveZ Sez - - I think the main message of the API breadth subsection is that - URLs, paths, etc would be modeled as formal types in Swift - (i.e. not as extensions on String). Second, I'd speculate less on - what Foundation could do (like extending String) and instead focus - on the fact that NSString still exists as an escape hatch for those - that feel that they need or want it. Furthermore, I'd move up the - "element access" discussion above the "escape hatch" discussion - (which should be last in the comparison with NSString discussion). - -API Breadth -~~~~~~~~~~~ - -The ``NSString`` interface clearly shows the effects of 20 years of -evolution through accretion. It is broad, with functionality -addressing encodings, paths, URLs, localization, and more. By -contrast, the interface to Swift's ``String`` is much narrower. - -.. _TBD: - -Of course, there's a reason for every ``NSString`` method, and the -full breadth of ``NSString`` functionality must remain accessible to -the Cocoa/Swift programmer. Fortunately, there are many ways to -address this need. For example: - -* The ``Foundation`` module can extend ``String`` with the methods of - ``NSString``. The extent to which we provide an identical-feeling - interface and/or correct any ``NSString`` misfeatures is still TBD - and wide open for discussion. - -* We can create a new modular interface in pure Swift, including a - ``Locale`` module that addresses localized string operations, an - ``Encoding`` module that addresses character encoding schemes, a - ``Regex`` module that provides regular expression functionality, - etc. Again, the specifics are TBD. - -* When all else fails, users can convert their Swift ``String``\ s to - ``NSString``\ s when they want to access ``NSString``-specific - functionality: - - .. parsed-literal:: - - **NString(mySwiftString)**\ .localizedStandardCompare(otherSwiftString) - -For Swift version 1.0, we err on the side of keeping the string -interface small, coherent, and sufficient for implementing -higher-level functionality. - -Element Access -~~~~~~~~~~~~~~ - -``NSString`` exposes UTF-16 `code units`__ as the primary element on -which indexing, slicing, and iteration operate. Swift's UTF-8 code -units are only available as a secondary interface. - -__ http://www.unicode.org/glossary/#code_unit - -``NSString`` is indexable and sliceable using ``Int``\ s, and so -exposes a ``length`` attribute. Swift's ``String`` is indexable and -sliceable using an abstract ``BidirectionalIndex`` type, and `does not -expose its length`__. - -__ length_ - -Sub-Strings -~~~~~~~~~~~ - -.. _range: - -Creating substrings in Swift is very fast. Therefore, Cocoa APIs that -operate on a substring given as an ``NSRange`` are replaced with Swift -APIs that just operate on ``String``\ s. One can use range-based -subscripting to achieve the same effect. For example: ``[str doFoo:arg -withRange:subrange]`` becomes ``str[subrange].doFoo(arg)``. - -``NSString`` Member-by-Member Comparison ----------------------------------------- - -:Notes: - * The following are from public headers from public frameworks, which - are AppKit and Foundation (verified). - - * Deprecated Cocoa APIs are not considered - - * A status of “*Remove*” below indicates a feature whose removal is - anticipated. Rationale is provided for these cases. - -Indexing -~~~~~~~~ - -.. _length: - ---------- - -.. sidebar:: Why doesn't ``String`` support ``.length``? - - In Swift, by convention, ``x.length`` is used to represent - the number of elements in a container, and since ``String`` is a - container of abstract |Character|_\ s, ``length`` would have to - count those. - - This meaning of ``length`` is unimplementable in O(1). It can be - cached, although not in the memory block where the characters are - stored, since we want a ``String`` to share storage with its - slices. Since the body of the ``String`` must already store the - ``String``\ 's *byte length*, caching the ``length`` would - increase the footprint of the top-level String object. Finally, - even if ``length`` were provided, doing things with ``String`` - that depend on a specific numeric ``length`` is error-prone. - -:Cocoa: - .. parsed-literal:: - - \- (NSUInteger)\ **length** - \- (unichar)\ **characterAtIndex:**\ (NSUInteger)index; - -:Swift: *not directly provided*, but similar functionality is - available: - - .. parsed-literal:: - - for j in 0...\ **s.bytes.length** { - doSomethingWith(**s.bytes[j]**) - } - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSRange)\ **rangeOfComposedCharacterSequenceAtIndex:**\ (NSUInteger)index; - \- (NSRange)\ **rangeOfComposedCharacterSequencesForRange:**\ (NSRange)range; - -:Swift: - .. parsed-literal:: - typealias IndexType = ... - func **indices**\ () -> Range - **subscript**\ (i: IndexType) -> Character - - .. Admonition:: Usage - - .. parsed-literal:: - - for i in someString.indices() { - doSomethingWith(\ **someString[i]**\ ) - } - - var (i,j) = **someString.indices().bounds** - while (i != j) { - doSomethingElseWith(\ **someString[i]**\ ) - ++i - } - - -Slicing -~~~~~~~ - -:Cocoa: - .. parsed-literal:: - \- (void)\ **getCharacters:**\ (unichar \*)\ **buffer range:**\ (NSRange)aRange; - -:Swift: - .. parsed-literal:: - typealias IndexType = ... - **subscript**\ (r: Range) -> Character - -Indexing -~~~~~~~~ - -:Cocoa: - .. parsed-literal:: - \- (NSString \*)\ **substringToIndex:**\ (NSUInteger)to; - \- (NSString \*)\ **substringFromIndex:**\ (NSUInteger)from; - \- (NSString \*)\ **substringWithRange:**\ (NSRange)range; - -:Swift: - .. parsed-literal:: - **subscript**\ (range : Range) -> String - - .. _slicing: - - .. Admonition:: Example - - .. parsed-literal:: - s[beginning...ending] // [s substringWithRange: NSMakeRange( beginning, ending )] - s[beginning...] // [s substringFromIndex: beginning] - s[...ending] // [s substringToIndex: ending] - - :Note: Swift may need additional interfaces to support - ``index...`` and ``...index`` notations. This part of the - ``Container`` protocol design isn't worked out yet. - -Comparison -~~~~~~~~~~~~ - -:Cocoa: - .. parsed-literal:: - \- (BOOL)\ **isEqualToString:**\ (NSString \*)aString; - \- (NSComparisonResult)\ **compare:**\ (NSString \*)string; - -:Swift: - .. parsed-literal:: - func **==** (lhs: String, rhs: String) -> Bool - func **!=** (lhs: String, rhs: String) -> Bool - func **<** (lhs: String, rhs: String) -> Bool - func **>** (lhs: String, rhs: String) -> Bool - func **<=** (lhs: String, rhs: String) -> Bool - func **>=** (lhs: String, rhs: String) -> Bool - -``NSString`` comparison is “literal” by default. As the documentation -says of ``isEqualToString``, - - “Ö” represented as the composed character sequence “O” and umlaut - would not compare equal to “Ö” represented as one Unicode character. - -By contrast, Swift string's primary comparison interface uses -Unicode's default collation_ algorithm, and is thus always -“Unicode-correct.” Unlike comparisons that depend on locale, it is -also stable across changes in system state. However, *just like* -``NSString``\ 's ``isEqualToString`` and ``compare`` methods, it -should not be expected to yield ideal (or even “proper”) results in -all contexts. - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSComparisonResult)\ **compare:**\ (NSString \*)string \ **options:**\ (NSStringCompareOptions)mask; - \- (NSComparisonResult)\ **compare:**\ (NSString \*)string \ **options:**\ (NSStringCompareOptions)mask \ **range:**\ (NSRange)compareRange; - \- (NSComparisonResult)\ **caseInsensitiveCompare:**\ (NSString \*)string; - -:Swift: *various compositions of primitive operations* / TBD_ - -* As noted above__, instead of passing sub-range arguments, we expect - Swift users to compose slicing_ with whole-string operations. - - __ range_ - -* Other details of these interfaces are distinguished by an - ``NSStringCompareOptions`` mask, of which - ``caseInsensitiveCompare:`` is essentially a special case: - - :``NSCaseInsensitiveSearch``: Whether a direct interface is needed - at all in Swift, and if so, its form, are TBD_. However, we - should consider following the lead of Python 3, wherein case - conversion also `normalizes letterforms`__. Then one can combine - ``String.toLower()`` with default comparison to get a - case-insensitive comparison:: - - { $0.toLower() == $1.toLower() } - - __ http://stackoverflow.com/a/11573384/125349 - - :``NSLiteralSearch``: Though it is the default for ``NSString``, - this option is essentially only useful as a performance - optimization when the string content is known to meet certain - restrictions (i.e. is known to be pure ASCII). When such - optimization is absolutely necessary, Swift standard library - algorithms can be used directly on the ``String``\ 's UTF8 code - units. However, Swift will also perform these optimizations - automatically (at the cost of a single test/branch) in many - cases, because each ``String`` stores a bit indicating whether - its content is known to be ASCII. - - :``NSBackwardsSearch``: It's unclear from the docs how this option - interacts with other ``NSString`` options, if at all, but basic - cases can be handled in Swift by ``s1.endsWith(s2)``. - - :``NSAnchoredSearch``: Not applicable to whole-string comparisons - :``NSNumericSearch``: While it's legitimate to defer this - functionality to Cocoa, it's (probably—see - ) locale-independent and - easy enough to implement in Swift. TBD_ - :``NSDiacriticInsensitiveSearch``: Ditto; TBD_ - :``NSWidthInsensitiveSearch``: Ditto; TBD_ - :``NSForcedOrderingSearch``: Ditto; TBD_. Also see - - :``NSRegularExpressionSearch``: We can defer this functionality to - Cocoa, or dispatch directly to ICU - as an optimization. It's unlikely - that we'll be building Swift its own - regexp engine for 1.0. - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSComparisonResult)\ **localizedCompare:**\ (NSString \*)string; - \- (NSComparisonResult)\ **localizedCaseInsensitiveCompare:**\ (NSString \*)string; - \- (NSComparisonResult)\ **localizedStandardCompare:**\ (NSString \*)string; - \- (NSComparisonResult)\ **compare:**\ (NSString \*)string \ **options:**\ (NSStringCompareOptions)mask \ **range:**\ (NSRange)compareRange \ **locale:**\ (id)locale; - -:Swift: As these all depend on locale, they are TBD_ - -Searching -~~~~~~~~~ - -.. Sidebar:: Rationale - - Modern languages (Java, C#, Python, Ruby…) have standardized on - variants of ``startsWith``/\ ``endsWith``. There's no reason Swift - should deviate from de-facto industry standards here. - -:Cocoa: - .. parsed-literal:: - \- (BOOL)\ **hasPrefix:**\ (NSString \*)aString; - \- (BOOL)\ **hasSuffix:**\ (NSString \*)aString; - -:Swift: - .. parsed-literal:: - func **startsWith**\ (prefix: String) - func **endsWith**\ (suffix: String) - ----- - -:Cocoa: - .. parsed-literal:: - \- (NSRange)\ **rangeOfString:**\ (NSString \*)aString; - -:Swift: - .. parsed-literal:: - func **find**\ (sought: String) -> Range - - .. Note:: Most other languages provide something like - ``s1.indexOf(s2)``, which returns only the starting index of - the first match. This is far less useful than the range of - the match, and is always available via - ``s1.find(s2).bounds.0`` - ----- - -:Cocoa: - .. parsed-literal:: - \- (NSRange)\ **rangeOfCharacterFromSet:**\ (NSCharacterSet \*)aSet; - -.. sidebar:: Naming - - The Swift function is just an algorithm that comes from conformance - to the ``Container`` protocol, which explains why it doesn't have a - ``String``\ -specific name. - -:Swift: - .. parsed-literal:: - func **find**\ (match: (Character)->Bool) -> Range - - .. Admonition:: Usage Example - - The ``NSString`` semantics can be achieved as follows: - - .. parsed-literal:: - - someString.find( {someCharSet.contains($0)} ) - ------ - -:Cocoa: - .. parsed-literal:: - \- (NSRange)\ **rangeOfString:**\ (NSString \*)aString \ **options:**\ (NSStringCompareOptions)mask; - \- (NSRange)\ **rangeOfString:**\ (NSString \*)aString \ **options:**\ (NSStringCompareOptions)mask \ **range:**\ (NSRange)searchRange; - \- (NSRange)\ **rangeOfString:**\ (NSString \*)aString \ **options:**\ (NSStringCompareOptions)mask \ **range:**\ (NSRange)searchRange \ **locale:**\ (NSLocale \*)locale; - - \- (NSRange)\ **rangeOfCharacterFromSet:**\ (NSCharacterSet \*)aSet \ **options:**\ (NSStringCompareOptions)mask; - \- (NSRange)\ **rangeOfCharacterFromSet:**\ (NSCharacterSet \*)aSet \ **options:**\ (NSStringCompareOptions)mask \ **range:**\ (NSRange)searchRange; - - These functions - -:Swift: *various compositions of primitive operations* / TBD_ - -Building -~~~~~~~~ - -:Cocoa: - .. parsed-literal:: - \- (NSString \*)\ **stringByAppendingString:**\ (NSString \*)aString; - -.. sidebar:: ``append`` - - the ``append`` method is a consequence of ``String``\ 's - conformance to ``OutputStream``. See the *Swift - formatting proposal* for details. - -:Swift: - .. parsed-literal:: - func **+** (lhs: String, rhs: String) -> String - func [infix,assignment] **+=** (lhs: [inout] String, rhs: String) - func **append**\ (suffix: String) - - -Dynamic Formatting -~~~~~~~~~~~~~~~~~~ - -:Cocoa: - .. parsed-literal:: - \- (NSString \*)\ **stringByAppendingFormat:**\ (NSString \*)format, ... NS_FORMAT_FUNCTION(1,2); - -:Swift: *Not directly provided*\ —see the *Swift formatting proposal* - -Extracting Numeric Values -~~~~~~~~~~~~~~~~~~~~~~~~~ - -:Cocoa: - .. parsed-literal:: - \- (double)doubleValue; - \- (float)floatValue; - \- (int)intValue; - \- (NSInteger)integerValue; - \- (long long)longLongValue; - \- (BOOL)boolValue; - -:Swift: Not in ``String``\ —It is up to other types to provide their - conversions to and from String. See also this `rationale`__ - - __ extending_ - -Splitting -~~~~~~~~~ - -:Cocoa: - .. parsed-literal:: - \- (NSArray \*)\ **componentsSeparatedByString:**\ (NSString \*)separator; - \- (NSArray \*)\ **componentsSeparatedByCharactersInSet:**\ (NSCharacterSet \*)separator; - -:Swift: - .. parsed-literal:: - func split(maxSplit: Int = Int.max()) -> String[] - func split(separator: Character, maxSplit: Int = Int.max()) -> String[] - - The semantics of these functions were taken from Python, which seems - to be a fairly good representative of what modern languages are - currently doing. The first overload splits on all whitespace - characters; the second only on specific characters. The universe of - possible splitting functions is quite broad, so the particulars of - this interface are **wide open for discussion**. In Swift right - now, these methods (on ``CodePoints``) are implemented in terms of a - generic algorithm: - - .. parsed-literal:: - - func **split**\ (seq: Seq, isSeparator: IsSeparator, maxSplit: Int = Int.max(), - allowEmptySlices: Bool = false ) -> Seq[] - -Splitting -~~~~~~~~~ - -:Cocoa: - .. parsed-literal:: - \- (NSString \*)\ **commonPrefixWithString:**\ (NSString \*)aString \ **options:**\ (NSStringCompareOptions)mask; - -:Swift: - .. parsed-literal:: - func **commonPrefix**\ (other: String) -> String - -Upper/Lowercase -~~~~~~~~~~~~~~~ - -:Cocoa: - .. parsed-literal:: - \- (NSString \*)\ **uppercaseString**; - \- (NSString \*)\ **uppercaseStringWithLocale:**\ (NSLocale \*)locale; - \- (NSString \*)\ **lowercaseString**; - \- (NSString \*)\ **lowercaseStringWithLocale:**\ (NSLocale \*)locale; - -.. sidebar:: Naming - - Other languages have overwhelmingly settled on ``upper()`` or - ``toUpper()`` for this functionality - -:Swift: - .. parsed-literal:: - func **toUpper**\ () -> String - func **toLower**\ () -> String - -Capitalization -~~~~~~~~~~~~~~ - -:Cocoa: - .. parsed-literal:: - \- (NSString \*)\ **capitalizedString**; - \- (NSString \*)\ **capitalizedStringWithLocale:**\ (NSLocale \*)locale; - -:Swift: - **TBD** - -.. Note:: ``NSString`` capitalizes the first letter of each substring - separated by spaces, tabs, or line terminators, which is in - no sense “Unicode-correct.” In most other languages that - support a ``capitalize`` method, it operates only on the - first character of the string, and capitalization-by-word is - named something like “``title``.” If Swift ``String`` - supports capitalization by word, it should be - Unicode-correct, but how we sort this particular area out is - still **TBD**. - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSString \*)\ **stringByTrimmingCharactersInSet:**\ (NSCharacterSet \*)set; - -:Swift: - .. parsed-literal:: - trim **trim**\ (match: (Character)->Bool) -> String - - .. Admonition:: Usage Example - - The ``NSString`` semantics can be achieved as follows: - - .. parsed-literal:: - - someString.trim( {someCharSet.contains($0)} ) - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSString \*)\ **stringByPaddingToLength:**\ (NSUInteger)newLength \ **withString:**\ (NSString \*)padString \ **startingAtIndex:**\ (NSUInteger)padIndex; - -:Swift: - .. parsed-literal:: *Not provided*. It's not clear whether this is - useful at all for non-ASCII strings, and - ---------- - -:Cocoa: - .. parsed-literal:: - \- (void)\ **getLineStart:**\ (NSUInteger \*)startPtr \ **end:**\ (NSUInteger \*)lineEndPtr \ **contentsEnd:**\ (NSUInteger \*)contentsEndPtr \ **forRange:**\ (NSRange)range; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSRange)\ **lineRangeForRange:**\ (NSRange)range; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (void)\ **getParagraphStart:**\ (NSUInteger \*)startPtr \ **end:**\ (NSUInteger \*)parEndPtr \ **contentsEnd:**\ (NSUInteger \*)contentsEndPtr \ **forRange:**\ (NSRange)range; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSRange)\ **paragraphRangeForRange:**\ (NSRange)range; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (void)\ **enumerateSubstringsInRange:**\ (NSRange)range \ **options:**\ (NSStringEnumerationOptions)opts \ **usingBlock:**\ (void (^)(NSString \*substring, NSRange substringRange, NSRange enclosingRange, BOOL \*stop))block; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (void)\ **enumerateLinesUsingBlock:**\ (void (^)(NSString \*line, BOOL \*stop))block; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSString \*)description; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSUInteger)hash; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSStringEncoding)fastestEncoding; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSStringEncoding)smallestEncoding; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSData \*)\ **dataUsingEncoding:**\ (NSStringEncoding)encoding \ **allowLossyConversion:**\ (BOOL)lossy; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSData \*)\ **dataUsingEncoding:**\ (NSStringEncoding)encoding; - -:Swift: - .. parsed-literal:: - **TBD** - -- (BOOL)\ **canBeConvertedToEncoding:**\ (NSStringEncoding)encoding; - - ---------- - -:Cocoa: - .. parsed-literal:: - \- (__strong const char \*)\ **cStringUsingEncoding:**\ (NSStringEncoding)encoding NS_RETURNS_INNER_POINTER; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (BOOL)\ **getCString:**\ (char \*)buffer \ **maxLength:**\ (NSUInteger)maxBufferCount \ **encoding:**\ (NSStringEncoding)encoding; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (BOOL)\ **getBytes:**\ (void \*)buffer \ **maxLength:**\ (NSUInteger)maxBufferCount \ **usedLength:**\ (NSUInteger \*)usedBufferCount \ **encoding:**\ (NSStringEncoding)encoding \ **options:**\ (NSStringEncodingConversionOptions)options \ **range:**\ (NSRange)range \ **remainingRange:**\ (NSRangePointer)leftover; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSUInteger)\ **maximumLengthOfBytesUsingEncoding:**\ (NSStringEncoding)enc; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSUInteger)\ **lengthOfBytesUsingEncoding:**\ (NSStringEncoding)enc; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSString \*)decomposedStringWithCanonicalMapping; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSString \*)precomposedStringWithCanonicalMapping; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSString \*)decomposedStringWithCompatibilityMapping; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSString \*)precomposedStringWithCompatibilityMapping; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSString \*)\ **stringByFoldingWithOptions:**\ (NSStringCompareOptions)options \ **locale:**\ (NSLocale \*)locale; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSString \*)\ **stringByReplacingOccurrencesOfString:**\ (NSString \*)target \ **withString:**\ (NSString \*)replacement \ **options:**\ (NSStringCompareOptions)options \ **range:**\ (NSRange)searchRange; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSString \*)\ **stringByReplacingOccurrencesOfString:**\ (NSString \*)target \ **withString:**\ (NSString \*)replacement; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (NSString \*)\ **stringByReplacingCharactersInRange:**\ (NSRange)range \ **withString:**\ (NSString \*)replacement; - - ---------- - -:Cocoa: - .. parsed-literal:: - \- (__strong const char \*)UTF8String NS_RETURNS_INNER_POINTER; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \+ (NSStringEncoding)defaultCStringEncoding; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \+ (const NSStringEncoding \*)availableStringEncodings; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \+ (NSString \*)\ **localizedNameOfStringEncoding:**\ (NSStringEncoding)encoding; - -Constructors -~~~~~~~~~~~~ - -:Cocoa: - .. parsed-literal:: - \- (instancetype)init; - ---------- - -:Cocoa: - .. parsed-literal:: - \- (instancetype)\ **initWithString:**\ (NSString \*)aString; - ---------- - -:Cocoa: - .. parsed-literal:: - \+ (instancetype)string; - ---------- - -:Cocoa: - .. parsed-literal:: - \+ (instancetype)\ **stringWithString:**\ (NSString \*)string; - -Not available (too error prone) - ---------- - -:Cocoa: - .. parsed-literal:: - \- (instancetype)\ **initWithCharactersNoCopy:**\ (unichar \*)characters \ **length:**\ (NSUInteger)length \ **freeWhenDone:**\ (BOOL)freeBuffer; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (instancetype)\ **initWithCharacters:**\ (const unichar \*)characters \ **length:**\ (NSUInteger)length; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (instancetype)\ **initWithUTF8String:**\ (const char \*)nullTerminatedCString; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (instancetype)\ **initWithFormat:**\ (NSString \*)format, ... NS_FORMAT_FUNCTION(1,2); - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (instancetype)\ **initWithFormat:**\ (NSString \*)format \ **arguments:**\ (va_list)argList NS_FORMAT_FUNCTION(1,0); - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (instancetype)\ **initWithFormat:**\ (NSString \*)format \ **locale:**\ (id)locale, ... NS_FORMAT_FUNCTION(1,3); - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (instancetype)\ **initWithFormat:**\ (NSString \*)format \ **locale:**\ (id)locale \ **arguments:**\ (va_list)argList NS_FORMAT_FUNCTION(1,0); - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (instancetype)\ **initWithData:**\ (NSData \*)data \ **encoding:**\ (NSStringEncoding)encoding; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (instancetype)\ **initWithBytes:**\ (const void \*)bytes \ **length:**\ (NSUInteger)len \ **encoding:**\ (NSStringEncoding)encoding; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (instancetype)\ **initWithBytesNoCopy:**\ (void \*)bytes \ **length:**\ (NSUInteger)len \ **encoding:**\ (NSStringEncoding)encoding \ **freeWhenDone:**\ (BOOL)freeBuffer; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \+ (instancetype)\ **stringWithCharacters:**\ (const unichar \*)characters \ **length:**\ (NSUInteger)length; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \+ (instancetype)\ **stringWithUTF8String:**\ (const char \*)nullTerminatedCString; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \+ (instancetype)\ **stringWithFormat:**\ (NSString \*)format, ... NS_FORMAT_FUNCTION(1,2); - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \+ (instancetype)\ **localizedStringWithFormat:**\ (NSString \*)format, ... NS_FORMAT_FUNCTION(1,2); - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \- (instancetype)\ **initWithCString:**\ (const char \*)nullTerminatedCString \ **encoding:**\ (NSStringEncoding)encoding; - -:Swift: - .. parsed-literal:: - **TBD** - ---------- - -:Cocoa: - .. parsed-literal:: - \+ (instancetype)\ **stringWithCString:**\ (const char \*)cString \ **encoding:**\ (NSStringEncoding)enc; - - -Linguistic Analysis -~~~~~~~~~~~~~~~~~~~ - -:Cocoa: - .. parsed-literal:: - \- (NSArray \*)\ **linguisticTagsInRange:**\ (NSRange)range \ **scheme:**\ (NSString \*)tagScheme \ **options:**\ (NSLinguisticTaggerOptions)opts \ **orthography:**\ (NSOrthography \*)orthography \ **tokenRanges:**\ (NSArray \*\*)tokenRanges; - \- (void)\ **enumerateLinguisticTagsInRange:**\ (NSRange)range \ **scheme:**\ (NSString \*)tagScheme \ **options:**\ (NSLinguisticTaggerOptions)opts \ **orthography:**\ (NSOrthography \*)orthography \ **usingBlock:**\ (void (^)(NSString \*tag, NSRange tokenRange, NSRange sentenceRange, BOOL \*stop))block; - -:Swift: - .. parsed-literal:: - **TBD** - -Unavailable on Swift Strings ----------------------------- - -URL Handling -~~~~~~~~~~~~ - -.. parsed-literal:: - - \- (instancetype)\ **initWithContentsOfURL:**\ (NSURL \*)url \ **encoding:**\ (NSStringEncoding)enc \ **error:**\ (NSError \*\*)error; - \+ (instancetype)\ **stringWithContentsOfURL:**\ (NSURL \*)url \ **encoding:**\ (NSStringEncoding)enc \ **error:**\ (NSError \*\*)error; - \- (instancetype)\ **initWithContentsOfURL:**\ (NSURL \*)url \ **usedEncoding:**\ (NSStringEncoding \*)enc \ **error:**\ (NSError \*\*)error; - \+ (instancetype)\ **stringWithContentsOfURL:**\ (NSURL \*)url \ **usedEncoding:**\ (NSStringEncoding \*)enc \ **error:**\ (NSError \*\*)error; - \- (BOOL)\ **writeToURL:**\ (NSURL \*)url \ **atomically:**\ (BOOL)useAuxiliaryFile \ **encoding:**\ (NSStringEncoding)enc \ **error:**\ (NSError \*\*)error; - \- (NSString \*)\ **stringByAddingPercentEncodingWithAllowedCharacters:**\ (NSCharacterSet \*)allowedCharacters; - \- (NSString \*)stringByRemovingPercentEncoding; - \- (NSString \*)\ **stringByAddingPercentEscapesUsingEncoding:**\ (NSStringEncoding)enc; - \- (NSString \*)\ **stringByReplacingPercentEscapesUsingEncoding:**\ (NSStringEncoding)enc; - -See: class File - -.. parsed-literal:: - - \- (instancetype)\ **initWithContentsOfFile:**\ (NSString \*)path \ **encoding:**\ (NSStringEncoding)enc \ **error:**\ (NSError \*\*)error; - \+ (instancetype)\ **stringWithContentsOfFile:**\ (NSString \*)path \ **encoding:**\ (NSStringEncoding)enc \ **error:**\ (NSError \*\*)error; - \- (instancetype)\ **initWithContentsOfFile:**\ (NSString \*)path \ **usedEncoding:**\ (NSStringEncoding \*)enc \ **error:**\ (NSError \*\*)error; - \+ (instancetype)\ **stringWithContentsOfFile:**\ (NSString \*)path \ **usedEncoding:**\ (NSStringEncoding \*)enc \ **error:**\ (NSError \*\*)error; - \- (BOOL)\ **writeToFile:**\ (NSString \*)path \ **atomically:**\ (BOOL)useAuxiliaryFile \ **encoding:**\ (NSStringEncoding)enc \ **error:**\ (NSError \*\*)error; - -Path Handling -~~~~~~~~~~~~~ - -.. parsed-literal:: - - \+ (NSString \*)\ **pathWithComponents:**\ (NSArray \*)components; - \- (NSArray \*)pathComponents; - \- (BOOL)isAbsolutePath; - \- (NSString \*)lastPathComponent; - \- (NSString \*)stringByDeletingLastPathComponent; - \- (NSString \*)\ **stringByAppendingPathComponent:**\ (NSString \*)str; - \- (NSString \*)pathExtension; - \- (NSString \*)stringByDeletingPathExtension; - \- (NSString \*)\ **stringByAppendingPathExtension:**\ (NSString \*)str; - \- (NSString \*)stringByAbbreviatingWithTildeInPath; - \- (NSString \*)stringByExpandingTildeInPath; - \- (NSString \*)stringByStandardizingPath; - \- (NSString \*)stringByResolvingSymlinksInPath; - \- (NSArray \*)\ **stringsByAppendingPaths:**\ (NSArray \*)paths; - \- (NSUInteger)\ **completePathIntoString:**\ (NSString \*\*)outputName \ **caseSensitive:**\ (BOOL)flag \ **matchesIntoArray:**\ (NSArray \*\*)outputArray \ **filterTypes:**\ (NSArray \*)filterTypes; - \- (__strong const char \*)fileSystemRepresentation NS_RETURNS_INNER_POINTER; - \- (BOOL)\ **getFileSystemRepresentation:**\ (char \*)cname \ **maxLength:**\ (NSUInteger)max; - -Property Lists -~~~~~~~~~~~~~~ - -Property lists are a feature of Cocoa. - -.. parsed-literal:: - - \- (id)propertyList; - \- (NSDictionary \*)propertyListFromStringsFileFormat; - Not applicable. Swift does not provide GUI support. - - \- (NSSize)\ **sizeWithAttributes:**\ (NSDictionary \*)attrs; - \- (void)\ **drawAtPoint:**\ (NSPoint)point \ **withAttributes:**\ (NSDictionary \*)attrs; - \- (void)\ **drawInRect:**\ (NSRect)rect \ **withAttributes:**\ (NSDictionary \*)attrs; - \- (void)\ **drawWithRect:**\ (NSRect)rect \ **options:**\ (NSStringDrawingOptions)options \ **attributes:**\ (NSDictionary \*)attributes; - \- (NSRect)\ **boundingRectWithSize:**\ (NSSize)size \ **options:**\ (NSStringDrawingOptions)options \ **attributes:**\ (NSDictionary \*)attributes; - \- (NSArray \*)\ **writableTypesForPasteboard:**\ (NSPasteboard \*)pasteboard; - \- (NSPasteboardWritingOptions)\ **writingOptionsForType:**\ (NSString \*)type \ **pasteboard:**\ (NSPasteboard \*)pasteboard; - \- (id)\ **pasteboardPropertyListForType:**\ (NSString \*)type; - \+ (NSArray \*)\ **readableTypesForPasteboard:**\ (NSPasteboard \*)pasteboard; - \+ (NSPasteboardReadingOptions)\ **readingOptionsForType:**\ (NSString \*)type \ **pasteboard:**\ (NSPasteboard \*)pasteboard; - \- (id)\ **initWithPasteboardPropertyList:**\ (id)propertyList \ **ofType:**\ (NSString \*)type; - -Deprecated APIs -~~~~~~~~~~~~~~~ - -Already deprecated in Cocoa. - -.. parsed-literal:: - - \- (const char \*)cString; - \- (const char \*)lossyCString; - \- (NSUInteger)cStringLength; - \- (void)\ **getCString:**\ (char \*)bytes; - \- (void)\ **getCString:**\ (char \*)bytes \ **maxLength:**\ (NSUInteger)maxLength; - \- (void)\ **getCString:**\ (char \*)bytes \ **maxLength:**\ (NSUInteger)maxLength \ **range:**\ (NSRange)aRange \ **remainingRange:**\ (NSRangePointer)leftoverRange; - \- (BOOL)\ **writeToFile:**\ (NSString \*)path \ **atomically:**\ (BOOL)useAuxiliaryFile; - \- (BOOL)\ **writeToURL:**\ (NSURL \*)url \ **atomically:**\ (BOOL)atomically; - \- (id)\ **initWithContentsOfFile:**\ (NSString \*)path; - \- (id)\ **initWithContentsOfURL:**\ (NSURL \*)url; - \+ (id)\ **stringWithContentsOfFile:**\ (NSString \*)path; - \+ (id)\ **stringWithContentsOfURL:**\ (NSURL \*)url; - \- (id)\ **initWithCStringNoCopy:**\ (char \*)bytes \ **length:**\ (NSUInteger)length \ **freeWhenDone:**\ (BOOL)freeBuffer; - \- (id)\ **initWithCString:**\ (const char \*)bytes \ **length:**\ (NSUInteger)length; - \- (id)\ **initWithCString:**\ (const char \*)bytes; - \+ (id)\ **stringWithCString:**\ (const char \*)bytes \ **length:**\ (NSUInteger)length; - \+ (id)\ **stringWithCString:**\ (const char \*)bytes; - \- (void)\ **getCharacters:**\ (unichar \*)buffer; - - --------------- - -Why YAGNI ---------- - -* Retroactive Modeling -* Derivation -* ... - -.. [#agnostic] Unicode specifies default (“un-tailored”) - locale-independent collation_ and segmentation_ algorithms that - make reasonable sense in most contexts. Using these algorithms - allows strings to be naturally compared and combined, generating - the expected results when the content is ASCII - -.. [#canonical] Technically, ``==`` checks for `Unicode canonical - equivalence`__ - - __ http://www.unicode.org/reports/tr15/tr15-18.html#Introduction - -.. [#locales] We have some specific ideas for locale-sensitive - interfaces, but details are still TBD and wide open for - discussion. - -.. [#re_sort] Collections that automatically re-sort based on locale - changes are out of scope for the core Swift language - -.. [#char] The type currently called ``Char`` in Swift represents a - Unicode code point. This document refers to it as ``CodePoint``, - in anticipation of renaming. - - -.. _segmentation: http://www.unicode.org/reports/tr29/#GB1 - -.. _collation: http://www.unicode.org/reports/tr10/ - - -.. [#code_points] When the user writes a string literal, she - specifies a particular sequence of code points. We guarantee that - those code points are stored without change in the resulting - ``String``. The user can explicitly request normalization, and - Swift can use a bit to remember whether a given string buffer has - been normalized, thus speeding up comparison operations. - -.. [#elements] Since ``String`` is locale-agnostic_, its elements are - determined using Unicode's default, “un-tailored” segmentation_ - algorithm. - diff --git a/docs/Testing.md b/docs/Testing.md new file mode 100644 index 0000000000000..eb74deab60c75 --- /dev/null +++ b/docs/Testing.md @@ -0,0 +1,344 @@ +Testing Swift +============= + +This document describes how we test the Swift compiler, the Swift +runtime, and the Swift standard library. + +Testing approaches +------------------ + +We use multiple approaches to test the Swift toolchain. + +- LLVM lit-based testsuites for the compiler, runtime and the + standard library. +- A selection of open source projects written in Swift. + +The LLVM lit-based testsuite +---------------------------- + +**Purpose**: primary testsuites for the Swift toolchain. + +**Contents**: Functional and regression tests for all toolchain +components. + +**Run by**: + +- Engineers and contributors are expected to run tests from these + testsuites locally before committing. (Usually on a single platform, + and not necessarily all tests.) +- Buildbots run all tests, on all supported platforms. + +### Running the LLVM lit-based testsuite + +You can run Swift tests using the `build-script`, or, alternatively, +using these targets in the build directory: + +- `check-swift` + + Runs tests from the `${SWIFT_SOURCE_ROOT}/test` directory. + +- `check-swift-validation` + + Runs tests from the + `${SWIFT_SOURCE_ROOT}/validation-test` directory. + +- `check-swift-all` + + Runs all tests. + +For day-to-day work on the Swift compiler, using check-swift should be +sufficient. The buildbot runs validation tests, so if those are +accidentally broken, it should not go unnoticed. + +Before committing a large change to a compiler (especially a language +change), or API changes to the standard library, it is recommended to +run validation test suite. + +For every target above, there are variants for different optimizations: + +- the target itself (e.g., `check-swift`) -- runs execution tests in + `-Onone` mode; +- the target with `-optimize` suffix (e.g., `check-swift-optimize`) -- + runs execution tests in `-O` mode; This target will only run tests + marked as `executable_test`. +- the target with `-optimize-unchecked` suffix (e.g., + `check-swift-optimize-unchecked`) -- runs execution tests in + `-Ounchecked` mode. This target will only run tests marked as + `executable_test`. + +If you need to manually run certain tests, you can invoke LLVM's lit.py +script directly. For example: + + % ${LLVM_SOURCE_ROOT}/utils/lit/lit.py -sv ${SWIFT_BUILD_ROOT}/test-iphonesimulator-i386/Parse/ + +This runs the tests in the test/Parse/ directory targeting the 32-bit +iOS Simulator. The `-sv` options give you a nice progress bar and only +show you output from the tests that fail. + +One downside of using this form is that you're appending relative paths +from the source directory to the test directory in your build directory. +(That is, there may not actually be a directory named 'Parse' in +'test-iphonesimulator-i386/'; the invocation works because there is one +in the source 'test/' directory.) There is a more verbose form that +specifies the testing configuration explicitly, which then allows you to +test files regardless of location. + + % ${LLVM_SOURCE_ROOT}/utils/lit/lit.py -sv --param swift_site_config=${SWIFT_BUILD_ROOT}/test-iphonesimulator-i386/lit.site.cfg ${SWIFT_SOURCE_ROOT}/test/Parse/ + +For more complicated configuration, copy the invocation from one of the +build targets mentioned above and modify it as necessary. lit.py also +has several useful features, like timing tests and providing a timeout. +Check these features out with `lit.py -h`. + +### Writing tests + +#### General guidelines + +When adding a new testcase, try to find an existing test file focused on +the same topic rather than starting a new test file. There is a fixed +runtime cost for every test file. On the other hand, avoid dumping new +tests in a file that is only remotely related to the purpose of the new +tests. + +Don't limit a test to a certain platform or hardware configuration just +because this makes the test slightly easier to write. This sometimes +means a little bit more work when adding the test, but the payoff from +the increased testing is significant. We heavily rely on portable tests +to port Swift to other platforms. + +Avoid using unstable language features in tests which test something +else (for example, avoid using an unstable underscored attribute when +another non-underscored attribute would work). + +Avoid using arbitrary implementation details of the standard library. +Always prefer to define types locally in the test, if feasible. + +Avoid purposefully shadowing names from the standard library, this makes +the test extremely confusing (if nothing else, to understand the intent +--- was the compiler bug triggered by this shadowing?) When reducing a +compiler testcase from the standard library source, rename the types and +APIs in the testcase to differ from the standard library APIs. + +In IRGen, SILGen and SIL tests, avoid using platform-dependent +implementation details of the standard library (unless doing so is point +of the test). Platform-dependent details include: + +- `Int` (use integer types with explicit types instead). +- Layout of `String`, `Array`, `Dictionary`, `Set`. These differ + between platforms that have Objective-C interop and those + that don't. + +Unless testing the standard library, avoid using arbitrary standard +library types and APIs, even if it is very convenient for you to do so +in your tests. Using the more common APIs like `Array` subscript or `+` +on `IntXX` is acceptable. This is important because you can't rely on +the full standard library being available. The long-term plan is to +introduce a mock, minimal standard library that only has a very basic +set of APIs. + +If you write an executable test please add `REQUIRES: executable_test` +to the test. + +#### Substitutions in lit tests + +Substitutions that start with `%target` configure the compiler for +building code for the target that is not the build machine: + +- `%target-parse-verify-swift`: parse and type check the current Swift + file for the target platform and verify diagnostics, like + `swift -parse -verify %s`. + + Use this substitution for testing semantic analysis in the compiler. + +- `%target-swift-frontend`: run `swift -frontend` for the target. + + Use this substitution (with extra arguments) for tests that don't + fit any other pattern. + +- `%target-swift-frontend(mock-sdk:` *mock sdk arguments* `)` *other + arguments*: like `%target-swift-frontend`, but allows to specify + command line parameters (typically `-sdk` and `-I`) to use a mock + SDK and SDK overlay that would take precedence over the target SDK. +- `%target-build-swift`: compile and link a Swift program for + the target. + + Use this substitution only when you intend to run the program later + in the test. + +- `%target-run-simple-swift`: build a one-file Swift program and run + it on the target machine. + + Use this substitution for executable tests that don't require + special compiler arguments. + + Add `REQUIRES: executable_test` to the test. + +- `%target-run-stdlib-swift`: like `%target-run-simple-swift` with + `-parse-stdlib -Xfrontend -disable-access-control`. + + This is sometimes useful for testing the Swift standard library. + + Add `REQUIRES: executable_test` to the test. + +- `%target-repl-run-simple-swift`: run a Swift program in a REPL on + the target machine. +- `%target-run`: run a command on the target machine. + + Add `REQUIRES: executable_test` to the test. + +- `%target-jit-run`: run a Swift program on the target machine using a + JIT compiler. +- `%target-swiftc_driver`: FIXME +- `%target-sil-opt`: run `sil-opt` for the target. +- `%target-sil-extract`: run `sil-extract` for the target. +- `%target-swift-ide-test`: run `swift-ide-test` for the target. +- `%target-swift-ide-test(mock-sdk:` *mock sdk arguments* `)` *other + arguments*: like `%target-swift-ide-test`, but allows to specify + command line parameters to use a mock SDK. +- `%target-swiftc_driver`: FIXME. +- `%target-swift-autolink-extract`: run `swift-autolink-extract` for + the target to extract its autolink flags on platforms that support + them (when the autolink-extract feature flag is set) +- `%target-clang`: run the system's `clang++` for the target. + + If you want to run the `clang` executable that was built alongside + Swift, use `%clang` instead. + +- `%target-ld`: run `ld` configured with flags pointing to the + standard library directory for the target. +- `%target-cc-options`: the clang flags to setup the target with the + right architecture and platform version. + +Always use `%target-*` substitutions unless you have a good reason. For +example, an exception would be a test that checks how the compiler +handles mixing module files for incompatible platforms (that test would +need to compile Swift code for two different platforms that are known to +be incompatible). + +When you can't use `%target-*` substitutions, you can use: + +- `%swift_driver_plain`: FIXME. +- `%swiftc_driver_plain`: FIXME. +- `%swift_driver`: FIXME. +- `%swiftc_driver`: FIXME. +- `%sil-opt`: FIXME. +- `%sil-extract`: FIXME. +- `%lldb-moduleimport-test`: FIXME. +- `%swift-ide-test_plain`: FIXME. +- `%swift-ide-test`: FIXME. +- `%llvm-opt`: FIXME. +- `%swift`: FIXME. +- `%clang-include-dir`: FIXME. +- `%clang-importer-sdk`: FIXME. + +Other substitutions: + +- `%leaks-runner`: FIXME. +- `%clang_apinotes`: FIXME. +- `%clang`: FIXME. +- `%target-triple`: FIXME, possible values. +- `%target-cpu`: FIXME, possible values. +- `%target-os`: FIXME, possible values. +- `%target-object-format`: the platform's object format (elf, + macho, coff). +- `%target-runtime`: the platform's Swift runtime (objc, native). +- `%target-ptrsize`: the pointer size of the target (32, 64). +- `%sdk`: FIXME. +- `%gyb`: FIXME. +- `%platform-module-dir`: absolute path of the directory where the + standard library module file for the target platform is stored. For + example, `/.../lib/swift/macosx`. +- `%platform-sdk-overlay-dir`: absolute path of the directory where + the SDK overlay module files for the target platform are stored. +- `%target-swiftmodule-name` and `%target-swiftdoc-name`: the basename + of swiftmodule and swiftdoc files for a framework compiled for the + target (for example, `arm64.swiftmodule` and `arm64.swiftdoc`). +- `%target-sdk-name`: only for Apple platforms: `xcrun`-style SDK name + (`macosx`, `iphoneos`, `iphonesimulator`). + +When writing a test where output (or IR, SIL) depends on the bitness of +the target CPU, use this pattern: + + // RUN: %target-swift-frontend ... | FileCheck --check-prefix=CHECK --check-prefix=CHECK-%target-ptrsize %s + + // CHECK: common line + // CHECK-32: only for 32-bit + // CHECK-64: only for 64-bit + + // FileCheck does a single pass for a combined set of CHECK lines, so you can + // do this: + // + // CHECK: define @foo() { + // CHECK-32: integer_literal $Builtin.Int32, 0 + // CHECK-64: integer_literal $Builtin.Int64, 0 + +When writing a test where output (or IR, SIL) depends on the target CPU +itself, use this pattern: + + // RUN: %target-swift-frontend ... | FileCheck --check-prefix=CHECK --check-prefix=CHECK-%target-cpu %s + + // CHECK: common line + // CHECK-i386: only for i386 + // CHECK-x86_64: only for x86_64 + // CHECK-armv7: only for armv7 + // CHECK-arm64: only for arm64 + +#### Features for `REQUIRES` and `XFAIL` + +FIXME: full list. + +- `swift_ast_verifier`: present if the AST verifier is enabled in + this build. + +When writing a test specific to x86, if possible, prefer +`REQUIRES: CPU=i386_or_x86_64` to `REQUIRES: CPU=x86_64`. + +`swift_test_mode_optimize[_unchecked|none]` and +`swift_test_mode_optimize[_unchecked|none]_` to specify a test +mode plus cpu configuration. + +`optimized_stdlib_`\` to specify a optimized stdlib plus cpu +configuration. + +#### Feature `REQUIRES: executable_test` + +This feature marks an executable test. The test harness makes this +feature generally available. It can be used to restrict the set of tests +to run. + +#### StdlibUnittest + +Tests accept command line parameters, run StdlibUnittest-based test +binary with `--help` for more information. + +#### Testing memory management in execution tests + +In execution tests, memory management testing should be performed using +local variables enclosed in a closure passed to the standard library +`autoreleasepool` function. For example: + + // A counter that's decremented by Canary's deinitializer. + var CanaryCount = 0 + + // A class whose instances increase a counter when they're destroyed. + class Canary { + deinit { ++CanaryCount } + } + + // Test that a local variable is correctly released before it goes out of + // scope. + CanaryCount = 0 + autoreleasepool { + let canary = Canary() + } + assert(CanaryCount == 1, "canary was not released") + +Memory management tests should be performed in a local scope because +Swift does not guarantee the destruction of global variables. Code that +needs to interoperate with Objective-C may put references in the +autorelease pool, so code that uses an `if true {}` or similar no-op +scope instead of `autoreleasepool` may falsely report leaks or fail to +catch overrelease bugs. If you're specifically testing the autoreleasing +behavior of code, or do not expect code to interact with the Objective-C +runtime, it may be OK to use `if true {}`, but those assumptions should +be commented in the test. diff --git a/docs/Testing.rst b/docs/Testing.rst deleted file mode 100644 index be91cbd822dc2..0000000000000 --- a/docs/Testing.rst +++ /dev/null @@ -1,367 +0,0 @@ -:orphan: - -.. @raise litre.TestsAreMissing - -============= -Testing Swift -============= - -This document describes how we test the Swift compiler, the Swift runtime, and -the Swift standard library. - -Testing approaches -================== - -We use multiple approaches to test the Swift toolchain. - -* LLVM lit-based testsuites for the compiler, runtime and the standard library. - -* A selection of open source projects written in Swift. - -The LLVM lit-based testsuite -============================ - -**Purpose**: primary testsuites for the Swift toolchain. - -**Contents**: Functional and regression tests for all toolchain components. - -**Run by**: - -* Engineers and contributors are expected to run tests from these testsuites - locally before committing. (Usually on a single platform, and not necessarily - all tests.) - -* Buildbots run all tests, on all supported platforms. - -Running the LLVM lit-based testsuite ------------------------------------- - -You can run Swift tests using the ``build-script``, or, alternatively, using -these targets in the build directory: - -* ``check-swift`` - - Runs tests from the ``${SWIFT_SOURCE_ROOT}/test`` directory. - -* ``check-swift-validation`` - - Runs tests from the ``${SWIFT_SOURCE_ROOT}/validation-test`` directory. - -* ``check-swift-all`` - - Runs all tests. - -For day-to-day work on the Swift compiler, using check-swift should be -sufficient. The buildbot runs validation tests, so if those are accidentally -broken, it should not go unnoticed. - -Before committing a large change to a compiler (especially a language change), -or API changes to the standard library, it is recommended to run validation -test suite. - -For every target above, there are variants for different optimizations: - -* the target itself (e.g., ``check-swift``) -- runs execution tests in - ``-Onone`` mode; - -* the target with ``-optimize`` suffix (e.g., ``check-swift-optimize``) -- runs - execution tests in ``-O`` mode; This target will only run tests marked as - ``executable_test``. - -* the target with ``-optimize-unchecked`` suffix (e.g., - ``check-swift-optimize-unchecked``) -- runs execution tests in - ``-Ounchecked`` mode. This target will only run tests marked as - ``executable_test``. - -If you need to manually run certain tests, you can invoke LLVM's lit.py script -directly. For example:: - - % ${LLVM_SOURCE_ROOT}/utils/lit/lit.py -sv ${SWIFT_BUILD_ROOT}/test-iphonesimulator-i386/Parse/ - -This runs the tests in the test/Parse/ directory targeting the 32-bit iOS -Simulator. The ``-sv`` options give you a nice progress bar and only show you -output from the tests that fail. - -One downside of using this form is that you're appending relative paths from -the source directory to the test directory in your build directory. (That is, -there may not actually be a directory named 'Parse' in -'test-iphonesimulator-i386/'; the invocation works because there is one in the -source 'test/' directory.) There is a more verbose form that specifies the -testing configuration explicitly, which then allows you to test files -regardless of location. - -:: - - % ${LLVM_SOURCE_ROOT}/utils/lit/lit.py -sv --param swift_site_config=${SWIFT_BUILD_ROOT}/test-iphonesimulator-i386/lit.site.cfg ${SWIFT_SOURCE_ROOT}/test/Parse/ - -For more complicated configuration, copy the invocation from one of the build -targets mentioned above and modify it as necessary. lit.py also has several -useful features, like timing tests and providing a timeout. Check these features -out with ``lit.py -h``. - -Writing tests -------------- - -General guidelines -^^^^^^^^^^^^^^^^^^ - -When adding a new testcase, try to find an existing test file focused on the -same topic rather than starting a new test file. There is a fixed runtime cost -for every test file. On the other hand, avoid dumping new tests in a file that -is only remotely related to the purpose of the new tests. - -Don't limit a test to a certain platform or hardware configuration just because -this makes the test slightly easier to write. This sometimes means a little -bit more work when adding the test, but the payoff from the increased testing -is significant. We heavily rely on portable tests to port Swift to other -platforms. - -Avoid using unstable language features in tests which test something else (for -example, avoid using an unstable underscored attribute when another -non-underscored attribute would work). - -Avoid using arbitrary implementation details of the standard library. Always -prefer to define types locally in the test, if feasible. - -Avoid purposefully shadowing names from the standard library, this makes the -test extremely confusing (if nothing else, to understand the intent --- was the -compiler bug triggered by this shadowing?) When reducing a compiler testcase -from the standard library source, rename the types and APIs in the testcase to -differ from the standard library APIs. - -In IRGen, SILGen and SIL tests, avoid using platform-dependent implementation -details of the standard library (unless doing so is point of the test). -Platform-dependent details include: - -* ``Int`` (use integer types with explicit types instead). - -* Layout of ``String``, ``Array``, ``Dictionary``, ``Set``. These differ - between platforms that have Objective-C interop and those that don't. - -Unless testing the standard library, avoid using arbitrary standard library -types and APIs, even if it is very convenient for you to do so in your tests. -Using the more common APIs like ``Array`` subscript or ``+`` on ``IntXX`` is -acceptable. This is important because you can't rely on the full standard -library being available. The long-term plan is to introduce a mock, minimal -standard library that only has a very basic set of APIs. - -If you write an executable test please add ``REQUIRES: executable_test`` to the -test. - -Substitutions in lit tests -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Substitutions that start with ``%target`` configure the compiler for building -code for the target that is not the build machine: - -* ``%target-parse-verify-swift``: parse and type check the current Swift file - for the target platform and verify diagnostics, like ``swift -parse -verify - %s``. - - Use this substitution for testing semantic analysis in the compiler. - -* ``%target-swift-frontend``: run ``swift -frontend`` for the target. - - Use this substitution (with extra arguments) for tests that don't fit any - other pattern. - -* ``%target-swift-frontend(mock-sdk:`` *mock sdk arguments* ``)`` *other - arguments*: like ``%target-swift-frontend``, but allows to specify command - line parameters (typically ``-sdk`` and ``-I``) to use a mock SDK and SDK - overlay that would take precedence over the target SDK. - -* ``%target-build-swift``: compile and link a Swift program for the target. - - Use this substitution only when you intend to run the program later in the - test. - -* ``%target-run-simple-swift``: build a one-file Swift program and run it on - the target machine. - - Use this substitution for executable tests that don't require special - compiler arguments. - - Add ``REQUIRES: executable_test`` to the test. - -* ``%target-run-stdlib-swift``: like ``%target-run-simple-swift`` with - ``-parse-stdlib -Xfrontend -disable-access-control``. - - This is sometimes useful for testing the Swift standard library. - - Add ``REQUIRES: executable_test`` to the test. - -* ``%target-repl-run-simple-swift``: run a Swift program in a REPL on the - target machine. - -* ``%target-run``: run a command on the target machine. - - Add ``REQUIRES: executable_test`` to the test. - -* ``%target-jit-run``: run a Swift program on the target machine using a JIT - compiler. - -* ``%target-swiftc_driver``: FIXME - -* ``%target-sil-opt``: run ``sil-opt`` for the target. - -* ``%target-sil-extract``: run ``sil-extract`` for the target. - -* ``%target-swift-ide-test``: run ``swift-ide-test`` for the target. - -* ``%target-swift-ide-test(mock-sdk:`` *mock sdk arguments* ``)`` *other - arguments*: like ``%target-swift-ide-test``, but allows to specify command - line parameters to use a mock SDK. - -* ``%target-swiftc_driver``: FIXME. - -* ``%target-swift-autolink-extract``: run ``swift-autolink-extract`` for the - target to extract its autolink flags on platforms that support them (when the - autolink-extract feature flag is set) - -* ``%target-clang``: run the system's ``clang++`` for the target. - - If you want to run the ``clang`` executable that was built alongside - Swift, use ``%clang`` instead. - -* ``%target-ld``: run ``ld`` configured with flags pointing to the standard - library directory for the target. - -* ``%target-cc-options``: the clang flags to setup the target with the right - architecture and platform version. - -Always use ``%target-*`` substitutions unless you have a good reason. For -example, an exception would be a test that checks how the compiler handles -mixing module files for incompatible platforms (that test would need to compile -Swift code for two different platforms that are known to be incompatible). - -When you can't use ``%target-*`` substitutions, you can use: - -* ``%swift_driver_plain``: FIXME. -* ``%swiftc_driver_plain``: FIXME. -* ``%swift_driver``: FIXME. -* ``%swiftc_driver``: FIXME. -* ``%sil-opt``: FIXME. -* ``%sil-extract``: FIXME. -* ``%lldb-moduleimport-test``: FIXME. -* ``%swift-ide-test_plain``: FIXME. -* ``%swift-ide-test``: FIXME. -* ``%llvm-opt``: FIXME. -* ``%swift``: FIXME. -* ``%clang-include-dir``: FIXME. -* ``%clang-importer-sdk``: FIXME. - -Other substitutions: - -* ``%leaks-runner``: FIXME. -* ``%clang_apinotes``: FIXME. -* ``%clang``: FIXME. -* ``%target-triple``: FIXME, possible values. -* ``%target-cpu``: FIXME, possible values. -* ``%target-os``: FIXME, possible values. -* ``%target-object-format``: the platform's object format (elf, macho, coff). -* ``%target-runtime``: the platform's Swift runtime (objc, native). -* ``%target-ptrsize``: the pointer size of the target (32, 64). -* ``%sdk``: FIXME. -* ``%gyb``: FIXME. - -* ``%platform-module-dir``: absolute path of the directory where the standard - library module file for the target platform is stored. For example, - ``/.../lib/swift/macosx``. - -* ``%platform-sdk-overlay-dir``: absolute path of the directory where the SDK - overlay module files for the target platform are stored. - -* ``%target-swiftmodule-name`` and ``%target-swiftdoc-name``: the basename of - swiftmodule and swiftdoc files for a framework compiled for the target (for - example, ``arm64.swiftmodule`` and ``arm64.swiftdoc``). - -* ``%target-sdk-name``: only for Apple platforms: ``xcrun``-style SDK name - (``macosx``, ``iphoneos``, ``iphonesimulator``). - -When writing a test where output (or IR, SIL) depends on the bitness of the -target CPU, use this pattern:: - - // RUN: %target-swift-frontend ... | FileCheck --check-prefix=CHECK --check-prefix=CHECK-%target-ptrsize %s - - // CHECK: common line - // CHECK-32: only for 32-bit - // CHECK-64: only for 64-bit - - // FileCheck does a single pass for a combined set of CHECK lines, so you can - // do this: - // - // CHECK: define @foo() { - // CHECK-32: integer_literal $Builtin.Int32, 0 - // CHECK-64: integer_literal $Builtin.Int64, 0 - -When writing a test where output (or IR, SIL) depends on the target CPU itself, -use this pattern:: - - // RUN: %target-swift-frontend ... | FileCheck --check-prefix=CHECK --check-prefix=CHECK-%target-cpu %s - - // CHECK: common line - // CHECK-i386: only for i386 - // CHECK-x86_64: only for x86_64 - // CHECK-armv7: only for armv7 - // CHECK-arm64: only for arm64 - -Features for ``REQUIRES`` and ``XFAIL`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -FIXME: full list. - -* ``swift_ast_verifier``: present if the AST verifier is enabled in this build. - -When writing a test specific to x86, if possible, prefer ``REQUIRES: -CPU=i386_or_x86_64`` to ``REQUIRES: CPU=x86_64``. - -``swift_test_mode_optimize[_unchecked|none]`` and -``swift_test_mode_optimize[_unchecked|none]_`` to specify a test mode -plus cpu configuration. - -``optimized_stdlib_``` to specify a optimized stdlib plus cpu -configuration. - -Feature ``REQUIRES: executable_test`` -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -This feature marks an executable test. The test harness makes this feature -generally available. It can be used to restrict the set of tests to run. - -StdlibUnittest -^^^^^^^^^^^^^^ - -Tests accept command line parameters, run StdlibUnittest-based test binary -with ``--help`` for more information. - -Testing memory management in execution tests -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -In execution tests, memory management testing should be performed -using local variables enclosed in a closure passed to the standard -library ``autoreleasepool`` function. For example:: - - // A counter that's decremented by Canary's deinitializer. - var CanaryCount = 0 - - // A class whose instances increase a counter when they're destroyed. - class Canary { - deinit { ++CanaryCount } - } - - // Test that a local variable is correctly released before it goes out of - // scope. - CanaryCount = 0 - autoreleasepool { - let canary = Canary() - } - assert(CanaryCount == 1, "canary was not released") - -Memory management tests should be performed in a local scope because Swift does -not guarantee the destruction of global variables. Code that needs to -interoperate with Objective-C may put references in the autorelease pool, so -code that uses an ``if true {}`` or similar no-op scope instead of -``autoreleasepool`` may falsely report leaks or fail to catch overrelease bugs. -If you're specifically testing the autoreleasing behavior of code, or do not -expect code to interact with the Objective-C runtime, it may be OK to use ``if -true {}``, but those assumptions should be commented in the test. diff --git a/docs/TextFormatting.md b/docs/TextFormatting.md new file mode 100644 index 0000000000000..dcfce827ce8b3 --- /dev/null +++ b/docs/TextFormatting.md @@ -0,0 +1,421 @@ +Text Formatting in Swift +======================== + +Author + +: Dave Abrahams + +Author + +: Chris Lattner + +Author + +: Dave Zarzycki + +Date + +: 2013-08-12 + +**Abstract:** We propose a system for creating textual representations +of Swift objects. Our system unifies conversion to `String`, string +interpolation, printing, and representation in the REPL and debugger. + +Scope +----- + +### Goals + +- The REPL and LLDB (“debuggers”) share formatting logic +- All types are “debug-printable” automatically +- Making a type “printable for humans” is super-easy +- `toString()`-ability is a consequence of printability. +- Customizing a type's printed representations is super-easy +- Format variations such as numeric radix are explicit and readable +- Large textual representations do not (necessarily) ever need to be + stored in memory, e.g. if they're being streamed into a file or over + a remote-debugging channel. + +### Non-Goals + +> **Rationale** +> +> Localization (including single-locale linguistic processing such as +> what's found in Clang's diagnostics subsystem) is the only major +> application we can think of for dynamically-constructed format +> strings, [^1] and is certainly the most important consumer of that +> feature. Therefore, localization and dynamic format strings should be +> designed together, and *under this proposal* the only format strings +> are string literals containing interpolations (“`\(...)`”). Cocoa +> programmers can still use Cocoa localization APIs for localization +> jobs. +> +> In Swift, only the most common cases need to be very terse. Anything +> “fancy” can afford to be a bit more verbose. If and when we address +> localization and design a full-featured dynamic string formatter, it +> may make sense to incorporate features of `printf` into the design. + +- **Localization** issues such as pluralizing and argument + presentation order are beyond the scope of this proposal. +- **Dynamic format strings** are beyond the scope of this proposal. +- **Matching the terseness of C**'s `printf` is a non-goal. + +CustomStringConvertible Types +----------------------------- + +`CustomStringConvertible` types can be used in string literal +interpolations, printed with `print(x)`, and can be converted to +`String` with `x.toString()`. + +The simple extension story for beginners is as follows: + +> “To make your type `CustomStringConvertible`, simply declare +> conformance to `CustomStringConvertible`: +> +> extension Person : CustomStringConvertible {} +> +> and it will have the same printed representation you see in the +> interpreter (REPL). To customize the representation, give your type a +> `func format()` that returns a `String`: +> +> extension Person : CustomStringConvertible { +> func format() -> String { +> return "\(lastName), \(firstName)" +> } +> } + +The formatting protocols described below allow more efficient and +flexible formatting as a natural extension of this simple story. + +Formatting Variants +------------------- + +`CustomStringConvertible` types with parameterized textual +representations (e.g. number types) *additionally* support a `format(…)` +method parameterized according to that type's axes of variability: + + print( offset ) + print( offset.format() ) // equivalent to previous line + print( offset.format(radix: 16, width: 5, precision: 3) ) + +Although `format(…)` is intended to provide the most general interface, +specialized formatting interfaces are also possible: + + print( offset.hex() ) + +Design Details +-------------- + +### Output Streams + +The most fundamental part of this design is `OutputStream`, a thing into +which we can stream text: [^2] + + protocol OutputStream { + func append(text: String) + } + +Every `String` can be used as an `OutputStream` directly: + + extension String : OutputStream { + func append(text: String) + } + +### Debug Printing + +Via compiler magic, *everything* conforms to the +`CustomDebugStringConvertible` protocol. To change the debug +representation for a type, you don't need to declare conformance: simply +give the type a `debugFormat()`: + + /// \brief A thing that can be printed in the REPL and the Debugger + protocol CustomDebugStringConvertible { + typealias DebugRepresentation : Streamable = String + + /// \brief Produce a textual representation for the REPL and + /// Debugger. + func debugFormat() -> DebugRepresentation + } + +Because `String` is a `Streamable`, your implementation of `debugFormat` +can just return a `String`. If want to write directly to the +`OutputStream` for efficiency reasons, (e.g. if your representation is +huge), you can return a custom `DebugRepresentation` type. + +> **Guideline** +> +> Producing a representation that can be consumed by the REPL and LLDB +> to produce an equivalent object is strongly encouraged where possible! +> For example, `String.debugFormat()` produces a representation starting +> and ending with “`"`”, where special characters are escaped, etc. A +> `struct Point { var x, y: Int }` might be represented as +> “`Point(x: 3, y: 5)`”. + +### (Non-Debug) Printing + +The `CustomStringConvertible` protocol provides a "pretty" textual +representation that can be distinct from the debug format. For example, +when `s` is a `String`, `s.format()` returns the string itself, without +quoting. + +Conformance to `CustomStringConvertible` is explicit, but if you want to +use the `debugFormat()` results for your type's `format()`, all you need +to do is declare conformance to `CustomStringConvertible`; there's +nothing to implement: + + /// \brief A thing that can be print()ed and toString()ed. + protocol CustomStringConvertible : CustomDebugStringConvertible { + typealias PrintRepresentation: Streamable = DebugRepresentation + + /// \brief produce a "pretty" textual representation. + /// + /// In general you can return a String here, but if you need more + /// control, return a custom Streamable type + func format() -> PrintRepresentation { + return debugFormat() + } + + /// \brief Simply convert to String + /// + /// You'll never want to reimplement this + func toString() -> String { + var result: String + self.format().write(result) + return result + } + } + +### `Streamable` + +Because it's not always efficient to construct a `String` representation +before writing an object to a stream, we provide a `Streamable` +protocol, for types that can write themselves into an `OutputStream`. +Every `Streamable` is also a `CustomStringConvertible`, naturally: + + protocol Streamable : CustomStringConvertible { + func writeTo(target: [inout] T) + + // You'll never want to reimplement this + func format() -> PrintRepresentation { + return this + } + } + +### How `String` Fits In + +`String`'s `debugFormat()` yields a `Streamable` that adds surrounding +quotes and escapes special characters: + + extension String : CustomDebugStringConvertible { + func debugFormat() -> EscapedStringRepresentation { + return EscapedStringRepresentation(self) + } + } + + struct EscapedStringRepresentation : Streamable { + var _value: String + + func writeTo(target: [inout] T) { + target.append("\"") + for c in _value { + target.append(c.escape()) + } + target.append("\"") + } + } + +Besides modeling `OutputStream`, `String` also conforms to `Streamable`: + + extension String : Streamable { + func writeTo(target: [inout] T) { + target.append(self) // Append yourself to the stream + } + + func format() -> String { + return this + } + } + +This conformance allows *most* formatting code to be written entirely in +terms of `String`, simplifying usage. Types with other needs can expose +lazy representations like `EscapedStringRepresentation` above. + +Extended Formatting Example +--------------------------- + +The following code is a scaled-down version of the formatting code used +for `Int`. It represents an example of how a relatively complicated +`format(…)` might be written: + + protocol CustomStringConvertibleInteger + : IntegerLiteralConvertible, Comparable, SignedNumber, CustomStringConvertible { + func %(lhs: Self, rhs: Self) -> Self + func /(lhs: Self, rhs: Self) -> Self + constructor(x: Int) + func toInt() -> Int + + func format(radix: Int = 10, fill: String = " ", width: Int = 0) + -> RadixFormat { + + return RadixFormat(this, radix: radix, fill: fill, width: width) + } + } + + struct RadixFormat : Streamable { + var value: T, radix = 10, fill = " ", width = 0 + + func writeTo(target: [inout] S) { + _writeSigned(value, &target) + } + + // Write the given positive value to stream + func _writePositive( + value: T, stream: [inout] S + ) -> Int { + if value == 0 { return 0 } + var radix: T = T.fromInt(self.radix) + var rest: T = value / radix + var nDigits = _writePositive(rest, &stream) + var digit = UInt32((value % radix).toInt()) + var baseCharOrd : UInt32 = digit <= 9 ? '0'.value : 'A'.value - 10 + stream.append(String(UnicodeScalar(baseCharOrd + digit))) + return nDigits + 1 + } + + func _writeSigned( + value: T, target: [inout] S + ) { + var width = 0 + var result = "" + + if value == 0 { + result = "0" + ++width + } + else { + var absVal = abs(value) + if (value < 0) { + target.append("-") + ++width + } + width += _writePositive(absVal, &result) + } + + while width < width { + ++width + target.append(fill) + } + target.append(result) + } + } + + extension Int : CustomStringConvertibleInteger { + func toInt() -> Int { return this } + } + +Possible Extensions (a.k.a. Complications) +------------------------------------------ + +We are not proposing these extensions. Since we have given them +considerable thought, they are included here for completeness and to +ensure our proposed design doesn't rule out important directions of +evolution. + +### `OutputStream` Adapters + +Most text transformations can be expressed as adapters over generic +`OutputStream`s. For example, it's easy to imagine an upcasing adapter +that transforms its input to upper case before writing it to an +underlying stream: + + struct UpperStream : OutputStream { + func append(x: String) { base.append( x.toUpper() ) } + var base: UnderlyingStream + } + +However, upcasing is a trivial example: many such transformations—such +as `trim()` or regex replacement—are stateful, which implies some way of +indicating “end of input” so that buffered state can be processed and +written to the underlying stream: + +This makes general `OutputStream` adapters more complicated to write and +use than ordinary `OutputStream`s. + +### `Streamable` Adapters + +For every conceivable `OutputStream` adaptor there's a corresponding +`Streamable` adaptor. For example: + + struct UpperStreamable { + var base: UnderlyingStreamable + + func writeTo(target: [inout] T) { + var adaptedStream = UpperStream(target) + self.base.writeTo(&adaptedStream) + target = adaptedStream.base + } + } + +Then, we could extend `Streamable` as follows: + + extension Streamable { + typealias Upcased : Streamable = UpperStreamable + func toUpper() -> UpperStreamable { + return Upcased(self) + } + } + +and, finally, we'd be able to write: + +The complexity of this back-and-forth adapter dance is daunting, and +might well be better handled in the language once we have some formal +model—such as coroutines—of inversion-of-control. We think it makes more +sense to build the important transformations directly into `format()` +methods, allowing, e.g.: + +Possible Simplifications +------------------------ + +One obvious simplification might be to fearlessly use `String` as the +universal textual representation type, rather than having a separate +`Streamable` protocol that doesn't necessarily create a fully-stored +representation. This approach would trade some efficiency for +considerable design simplicity. It is reasonable to ask whether the +efficiency cost would be significant in real cases, and the truth is +that we don't have enough information to know. At least until we do, we +opt not to trade away any CPU, memory, and power. + +If we were willing to say that only `class`es can conform to +`OutputStream`, we could eliminate the explicit `[inout]` where +`OutputStream`s are passed around. Then, we'd simply need a +`class StringStream` for creating `String` representations. It would +also make `OutputStream` adapters a *bit* simpler to use because you'd +never need to “write back” explicitly onto the target stream. However, +stateful `OutputStream` adapters would still need a `close()` method, +which makes a perfect place to return a copy of the underlying stream, +which can then be “written back”: + +We think anyone writing such adapters can handle the need for explicit +write-back, and the ability to use `String` as an `OutputStream` without +additionally allocating a `StringStream` on the heap seems to tip the +balance in favor of the current design. + +------------------------------------------------------------------------ + +[^1]: In fact it's possible to imagine a workable system for + localization that does away with dynamic format strings altogether, + so that all format strings are fully statically-checked and some of + the same formatting primitives can be used by localizers as by + fully-privileged Swift programmers. This approach would involve + compiling/JIT-ing localizations into dynamically-loaded modules. In + any case, that will wait until we have native Swift dylibs. + +[^2]: We don't support streaming individual code points directly because + it's possible to create invalid sequences of code points. For any + code point that, on its own, represents a valid `Character` (a.k.a. + Unicode extended grapheme cluster\_\_), it is trivial and + inexpensive to create a `String`. For more information on the + relationship between `String` and `Character` see the (forthcoming, + as of this writing) document *Swift Strings State of the Union*. + + \_\_ diff --git a/docs/TextFormatting.rst b/docs/TextFormatting.rst deleted file mode 100644 index 6c9bedfdc8ce7..0000000000000 --- a/docs/TextFormatting.rst +++ /dev/null @@ -1,477 +0,0 @@ -:orphan: - -Text Formatting in Swift -======================== - -:Author: Dave Abrahams -:Author: Chris Lattner -:Author: Dave Zarzycki -:Date: 2013-08-12 - - -.. contents:: Index - -**Abstract:** We propose a system for creating textual representations -of Swift objects. Our system unifies conversion to ``String``, string -interpolation, printing, and representation in the REPL and debugger. - -Scope ------ - -Goals -..... - -* The REPL and LLDB (“debuggers”) share formatting logic -* All types are “debug-printable” automatically -* Making a type “printable for humans” is super-easy -* ``toString()``-ability is a consequence of printability. -* Customizing a type's printed representations is super-easy -* Format variations such as numeric radix are explicit and readable -* Large textual representations do not (necessarily) ever need to be - stored in memory, e.g. if they're being streamed into a file or over - a remote-debugging channel. - -Non-Goals -......... - -.. sidebar:: Rationale - - Localization (including single-locale linguistic processing such as - what's found in Clang's diagnostics subsystem) is the only major - application we can think of for dynamically-constructed format - strings, [#dynamic]_ and is certainly the most important consumer of - that feature. Therefore, localization and dynamic format strings - should be designed together, and *under this proposal* the only - format strings are string literals containing interpolations - (“``\(...)``”). Cocoa programmers can still use Cocoa localization - APIs for localization jobs. - - In Swift, only the most common cases need to be very terse. - Anything “fancy” can afford to be a bit more verbose. If and when - we address localization and design a full-featured dynamic string - formatter, it may make sense to incorporate features of ``printf`` - into the design. - -* **Localization** issues such as pluralizing and argument - presentation order are beyond the scope of this proposal. - -* **Dynamic format strings** are beyond the scope of this proposal. - -* **Matching the terseness of C**\ 's ``printf`` is a non-goal. - -CustomStringConvertible Types ------------------------------ - -``CustomStringConvertible`` types can be used in string literal interpolations, -printed with ``print(x)``, and can be converted to ``String`` with -``x.toString()``. - -The simple extension story for beginners is as follows: - - “To make your type ``CustomStringConvertible``, simply declare conformance to - ``CustomStringConvertible``:: - - extension Person : CustomStringConvertible {} - - and it will have the same printed representation you see in the - interpreter (REPL). To customize the representation, give your type - a ``func format()`` that returns a ``String``:: - - extension Person : CustomStringConvertible { - func format() -> String { - return "\(lastName), \(firstName)" - } - } - -The formatting protocols described below allow more efficient and -flexible formatting as a natural extension of this simple story. - -Formatting Variants -------------------- - -``CustomStringConvertible`` types with parameterized textual representations -(e.g. number types) *additionally* support a ``format(…)`` method -parameterized according to that type's axes of variability:: - - print( offset ) - print( offset.format() ) // equivalent to previous line - print( offset.format(radix: 16, width: 5, precision: 3) ) - -Although ``format(…)`` is intended to provide the most general -interface, specialized formatting interfaces are also possible:: - - print( offset.hex() ) - - -Design Details --------------- - -Output Streams -.............. - -The most fundamental part of this design is ``OutputStream``, a thing -into which we can stream text: [#character1]_ - -:: - - protocol OutputStream { - func append(text: String) - } - -Every ``String`` can be used as an ``OutputStream`` directly:: - - extension String : OutputStream { - func append(text: String) - } - -Debug Printing -.............. - -Via compiler magic, *everything* conforms to the ``CustomDebugStringConvertible`` -protocol. To change the debug representation for a type, you don't -need to declare conformance: simply give the type a ``debugFormat()``:: - - /// \brief A thing that can be printed in the REPL and the Debugger - protocol CustomDebugStringConvertible { - typealias DebugRepresentation : Streamable = String - - /// \brief Produce a textual representation for the REPL and - /// Debugger. - func debugFormat() -> DebugRepresentation - } - -Because ``String`` is a ``Streamable``, your implementation of -``debugFormat`` can just return a ``String``. If want to write -directly to the ``OutputStream`` for efficiency reasons, -(e.g. if your representation is huge), you can return a custom -``DebugRepresentation`` type. - - -.. Admonition:: Guideline - - Producing a representation that can be consumed by the REPL - and LLDB to produce an equivalent object is strongly encouraged - where possible! For example, ``String.debugFormat()`` produces - a representation starting and ending with “``"``”, where special - characters are escaped, etc. A ``struct Point { var x, y: Int }`` - might be represented as “``Point(x: 3, y: 5)``”. - -(Non-Debug) Printing -.................... - -The ``CustomStringConvertible`` protocol provides a "pretty" textual representation -that can be distinct from the debug format. For example, when ``s`` -is a ``String``, ``s.format()`` returns the string itself, -without quoting. - -Conformance to ``CustomStringConvertible`` is explicit, but if you want to use the -``debugFormat()`` results for your type's ``format()``, all you -need to do is declare conformance to ``CustomStringConvertible``; there's nothing to -implement:: - - /// \brief A thing that can be print()ed and toString()ed. - protocol CustomStringConvertible : CustomDebugStringConvertible { - typealias PrintRepresentation: Streamable = DebugRepresentation - - /// \brief produce a "pretty" textual representation. - /// - /// In general you can return a String here, but if you need more - /// control, return a custom Streamable type - func format() -> PrintRepresentation { - return debugFormat() - } - - /// \brief Simply convert to String - /// - /// You'll never want to reimplement this - func toString() -> String { - var result: String - self.format().write(result) - return result - } - } - -``Streamable`` -............... - -Because it's not always efficient to construct a ``String`` -representation before writing an object to a stream, we provide a -``Streamable`` protocol, for types that can write themselves into an -``OutputStream``. Every ``Streamable`` is also a ``CustomStringConvertible``, -naturally:: - - protocol Streamable : CustomStringConvertible { - func writeTo(target: [inout] T) - - // You'll never want to reimplement this - func format() -> PrintRepresentation { - return this - } - } - -How ``String`` Fits In -...................... - -``String``\ 's ``debugFormat()`` yields a ``Streamable`` that -adds surrounding quotes and escapes special characters:: - - extension String : CustomDebugStringConvertible { - func debugFormat() -> EscapedStringRepresentation { - return EscapedStringRepresentation(self) - } - } - - struct EscapedStringRepresentation : Streamable { - var _value: String - - func writeTo(target: [inout] T) { - target.append("\"") - for c in _value { - target.append(c.escape()) - } - target.append("\"") - } - } - -Besides modeling ``OutputStream``, ``String`` also conforms to -``Streamable``:: - - extension String : Streamable { - func writeTo(target: [inout] T) { - target.append(self) // Append yourself to the stream - } - - func format() -> String { - return this - } - } - -This conformance allows *most* formatting code to be written entirely -in terms of ``String``, simplifying usage. Types with other needs can -expose lazy representations like ``EscapedStringRepresentation`` -above. - -Extended Formatting Example ---------------------------- - -The following code is a scaled-down version of the formatting code -used for ``Int``. It represents an example of how a relatively -complicated ``format(…)`` might be written:: - - protocol CustomStringConvertibleInteger - : IntegerLiteralConvertible, Comparable, SignedNumber, CustomStringConvertible { - func %(lhs: Self, rhs: Self) -> Self - func /(lhs: Self, rhs: Self) -> Self - constructor(x: Int) - func toInt() -> Int - - func format(radix: Int = 10, fill: String = " ", width: Int = 0) - -> RadixFormat { - - return RadixFormat(this, radix: radix, fill: fill, width: width) - } - } - - struct RadixFormat : Streamable { - var value: T, radix = 10, fill = " ", width = 0 - - func writeTo(target: [inout] S) { - _writeSigned(value, &target) - } - - // Write the given positive value to stream - func _writePositive( - value: T, stream: [inout] S - ) -> Int { - if value == 0 { return 0 } - var radix: T = T.fromInt(self.radix) - var rest: T = value / radix - var nDigits = _writePositive(rest, &stream) - var digit = UInt32((value % radix).toInt()) - var baseCharOrd : UInt32 = digit <= 9 ? '0'.value : 'A'.value - 10 - stream.append(String(UnicodeScalar(baseCharOrd + digit))) - return nDigits + 1 - } - - func _writeSigned( - value: T, target: [inout] S - ) { - var width = 0 - var result = "" - - if value == 0 { - result = "0" - ++width - } - else { - var absVal = abs(value) - if (value < 0) { - target.append("-") - ++width - } - width += _writePositive(absVal, &result) - } - - while width < width { - ++width - target.append(fill) - } - target.append(result) - } - } - - extension Int : CustomStringConvertibleInteger { - func toInt() -> Int { return this } - } - - -Possible Extensions (a.k.a. Complications) ------------------------------------------- - -We are not proposing these extensions. Since we have given them -considerable thought, they are included here for completeness and to -ensure our proposed design doesn't rule out important directions of -evolution. - -``OutputStream`` Adapters -......................... - -Most text transformations can be expressed as adapters over generic -``OutputStream``\ s. For example, it's easy to imagine an upcasing -adapter that transforms its input to upper case before writing it to -an underlying stream:: - - struct UpperStream : OutputStream { - func append(x: String) { base.append( x.toUpper() ) } - var base: UnderlyingStream - } - -However, upcasing is a trivial example: many such transformations—such -as ``trim()`` or regex replacement—are stateful, which implies some -way of indicating “end of input” so that buffered state can be -processed and written to the underlying stream: - -.. parsed-literal:: - - struct TrimStream : OutputStream { - func append(x: String) { ... } - **func close() { ... }** - var base: UnderlyingStream - var bufferedWhitespace: String - } - -This makes general ``OutputStream`` adapters more complicated to write -and use than ordinary ``OutputStream``\ s. - -``Streamable`` Adapters -....................... - -For every conceivable ``OutputStream`` adaptor there's a corresponding -``Streamable`` adaptor. For example:: - - struct UpperStreamable { - var base: UnderlyingStreamable - - func writeTo(target: [inout] T) { - var adaptedStream = UpperStream(target) - self.base.writeTo(&adaptedStream) - target = adaptedStream.base - } - } - -Then, we could extend ``Streamable`` as follows:: - - extension Streamable { - typealias Upcased : Streamable = UpperStreamable - func toUpper() -> UpperStreamable { - return Upcased(self) - } - } - -and, finally, we'd be able to write: - -.. parsed-literal:: - - print( n.format(radix:16)\ **.toUpper()** ) - -The complexity of this back-and-forth adapter dance is daunting, and -might well be better handled in the language once we have some formal -model—such as coroutines—of inversion-of-control. We think it makes -more sense to build the important transformations directly into -``format()`` methods, allowing, e.g.: - -.. parsed-literal:: - - print( n.format(radix:16, **case:.upper** ) ) - -Possible Simplifications ------------------------- - -One obvious simplification might be to fearlessly use ``String`` as -the universal textual representation type, rather than having a -separate ``Streamable`` protocol that doesn't necessarily create a -fully-stored representation. This approach would trade some -efficiency for considerable design simplicity. It is reasonable to -ask whether the efficiency cost would be significant in real cases, -and the truth is that we don't have enough information to know. At -least until we do, we opt not to trade away any CPU, memory, and -power. - -If we were willing to say that only ``class``\ es can conform to -``OutputStream``, we could eliminate the explicit ``[inout]`` where -``OutputStream``\ s are passed around. Then, we'd simply need a -``class StringStream`` for creating ``String`` representations. It -would also make ``OutputStream`` adapters a *bit* simpler to use -because you'd never need to “write back” explicitly onto the target -stream. However, stateful ``OutputStream`` adapters would still need a -``close()`` method, which makes a perfect place to return a copy of -the underlying stream, which can then be “written back”: - -.. parsed-literal:: - - struct AdaptedStreamable { - ... - func writeTo(target: [inout] Target) { - // create the stream that transforms the representation - var adaptedTarget = adapt(target, adapter); - // write the Base object to the target stream - base.writeTo(&adaptedTarget) - // Flush the adapted stream and, in case Target is a value type, - // write its new value - **target = adaptedTarget.close()** - } - ... - } - -We think anyone writing such adapters can handle the need for explicit -write-back, and the ability to use ``String`` as an ``OutputStream`` -without additionally allocating a ``StringStream`` on the heap seems -to tip the balance in favor of the current design. - --------- - -.. [#format] Whether ``format(…)`` is to be a real protocol or merely - an ad-hoc convention is TBD. So far, there's no obvious use for a - generic ``format`` with arguments that depend on the type being - formatted, so an ad-hoc convention would be just fine. - -.. [#character1] We don't support streaming individual code points - directly because it's possible to create invalid sequences of code - points. For any code point that, on its own, represents a valid - ``Character`` (a.k.a. Unicode `extended grapheme cluster`__), it is - trivial and inexpensive to create a ``String``. For more - information on the relationship between ``String`` and - ``Character`` see the (forthcoming, as of this writing) document - *Swift Strings State of the Union*. - - __ http://www.unicode.org/glossary/#extended_grapheme_cluster - -.. [#dynamic] In fact it's possible to imagine a workable system for - localization that does away with dynamic format strings altogether, - so that all format strings are fully statically-checked and some of - the same formatting primitives can be used by localizers as by - fully-privileged Swift programmers. This approach would involve - compiling/JIT-ing localizations into dynamically-loaded modules. - In any case, that will wait until we have native Swift dylibs. - - diff --git a/docs/TransparentAttr.md b/docs/TransparentAttr.md new file mode 100644 index 0000000000000..f0dcdba73a3c1 --- /dev/null +++ b/docs/TransparentAttr.md @@ -0,0 +1,71 @@ +`@_transparent` +=============== + +Semantically, `@_transparent` means something like "treat this operation +as if it were a primitive operation". The name is meant to imply that +both the compiler and the compiled program will "see through" the +operation to its implementation. + +This has several consequences: + +- Any calls to a function marked `@_transparent` MUST be inlined prior + to doing dataflow-related diagnostics, even under `-Onone`. This may + be necessary to *catch* dataflow errors. +- Because of this, a `@_transparent` function is inherently "fragile", + in that changing its implementation most likely will not affect + callers in existing compiled binaries. +- Because of this, a `@_transparent` function MUST only reference + public symbols, and MUST not be optimized based on knowledge of the + module it's in. \[This is not currently implemented or enforced.\] +- Debug info SHOULD skip over the inlined operations when + single-stepping through the calling function. + +This is all that `@_transparent` means. + +When should you use `@_transparent`? +------------------------------------ + +- Does the implementation of this function ever have to change? Then + you can't allow it to be inlined. +- Does the implementation need to call private things---either + true-`private` functions, or `internal` functions that might go away + in the next release? Then you can't allow it to be inlined. (Well, + you can for now for `internal`, but it'll break once we have + libraries that aren't shipped with apps.) +- Is it okay if the function is *not* inlined? You'd just prefer that + it were? Then you should use \[the attribute we haven't designed + yet\], rather than `@_transparent`. (If you really need this right + now, try `@inline(__always)`.) +- Is it a problem if the function is inlined even under `-Onone`? Then + you're really in the previous case. Trust the compiler. +- Is it a problem if you can't step through the function that's been + inlined? Then you don't want `@_transparent`; you just want + `@inline(__always)`. +- Is it okay if the inlining happens after all the dataflow + diagnostics? Then you don't want `@_transparent`; you just want + `@inline(__always)`. + +If you made it this far, it sounds like `@_transparent` is the right +choice. + +Current implementation limitations +---------------------------------- + +- We don't have a general `@inlineable` attribute for functions that + *allows* inlining but doesn't *require* it. +- As mentioned above, we don't enforce that inlineable things only + refer to public symbols. rdar://problem/22666548 +- We also don't keep from optimizing based on implementation details + of the current module. \[No Radar yet.\] +- If you have local types in your inlineable function, serialization + falls over. (As does textual SIL.) rdar://problem/17631278 +- When compiling in non-single-frontend mode, SIL is generated for + each file but then thrown away in the "merge modules" step. So none + of it is inlineable for external callers. (Currently, + `-whole-module-optimization` is equivalent to + `-force-single-frontend-invocation`.) rdar://problem/18913977 +- Similarly, when compiling in non-single-frontend mode, no SIL is + generated for any functions but those in the primary file (for each + frontend invocation), including `@inline(__always)` and + `@_transparent` functions. This is semantically a bug. + rdar://problem/15366167 diff --git a/docs/TransparentAttr.rst b/docs/TransparentAttr.rst deleted file mode 100644 index 4b738c0cf3694..0000000000000 --- a/docs/TransparentAttr.rst +++ /dev/null @@ -1,83 +0,0 @@ -:orphan: - -``@_transparent`` -================= - -Semantically, ``@_transparent`` means something like "treat this operation as -if it were a primitive operation". The name is meant to imply that both the -compiler and the compiled program will "see through" the operation to its -implementation. - -This has several consequences: - -- Any calls to a function marked ``@_transparent`` MUST be inlined prior to - doing dataflow-related diagnostics, even under ``-Onone``. This may be - necessary to *catch* dataflow errors. - -- Because of this, a ``@_transparent`` function is inherently "fragile", in - that changing its implementation most likely will not affect callers in - existing compiled binaries. - -- Because of this, a ``@_transparent`` function MUST only reference public - symbols, and MUST not be optimized based on knowledge of the module it's in. - [This is not currently implemented or enforced.] - -- Debug info SHOULD skip over the inlined operations when single-stepping - through the calling function. - -This is all that ``@_transparent`` means. - - -When should you use ``@_transparent``? --------------------------------------- - -- Does the implementation of this function ever have to change? Then you can't - allow it to be inlined. - -- Does the implementation need to call private things---either true-``private`` - functions, or ``internal`` functions that might go away in the next release? - Then you can't allow it to be inlined. (Well, you can for now for - ``internal``, but it'll break once we have libraries that aren't shipped with - apps.) - -- Is it okay if the function is *not* inlined? You'd just prefer that it were? - Then you should use [the attribute we haven't designed yet], rather than - ``@_transparent``. (If you really need this right now, try - ``@inline(__always)``.) - -- Is it a problem if the function is inlined even under ``-Onone``? Then you're - really in the previous case. Trust the compiler. - -- Is it a problem if you can't step through the function that's been inlined? - Then you don't want ``@_transparent``; you just want ``@inline(__always)``. - -- Is it okay if the inlining happens after all the dataflow diagnostics? Then - you don't want ``@_transparent``; you just want ``@inline(__always)``. - -If you made it this far, it sounds like ``@_transparent`` is the right choice. - - -Current implementation limitations ----------------------------------- - -- We don't have a general ``@inlineable`` attribute for functions that *allows* - inlining but doesn't *require* it. - -- As mentioned above, we don't enforce that inlineable things only refer to - public symbols. rdar://problem/22666548 - -- We also don't keep from optimizing based on implementation details of the - current module. [No Radar yet.] - -- If you have local types in your inlineable function, serialization falls - over. (As does textual SIL.) rdar://problem/17631278 - -- When compiling in non-single-frontend mode, SIL is generated for each file - but then thrown away in the "merge modules" step. So none of it is inlineable - for external callers. (Currently, ``-whole-module-optimization`` is - equivalent to ``-force-single-frontend-invocation``.) rdar://problem/18913977 - -- Similarly, when compiling in non-single-frontend mode, no SIL is generated for - any functions but those in the primary file (for each frontend invocation), - including ``@inline(__always)`` and ``@_transparent`` functions. This is - semantically a bug. rdar://problem/15366167 diff --git a/docs/TypeChecker.md b/docs/TypeChecker.md new file mode 100644 index 0000000000000..a16a3d2c90488 --- /dev/null +++ b/docs/TypeChecker.md @@ -0,0 +1,977 @@ +Type Checker Design and Implementation +====================================== + +Purpose +------- + +This document describes the design and implementation of the Swift type +checker. It is intended for developers who wish to modify, extend, or +improve on the type checker, or simply to understand in greater depth +how the Swift type system works. Familiarity with the Swift programming +language is assumed. + +Approach +-------- + +The Swift language and its type system incorporate a number of popular +language features, including object-oriented programming via classes, +function and operator overloading, subtyping, and constrained parametric +polymorphism. Swift makes extensive use of type inference, allowing one +to omit the types of many variables and expressions. For example: + + func round(x: Double) -> Int { /* ... */ } + var pi: Double = 3.14159 + var three = round(pi) // 'three' has type 'Int' + + func identity(x: T) -> T { return x } + var eFloat: Float = -identity(2.71828) // numeric literal gets type 'Float' + +Swift's type inference allows type information to flow in two +directions. As in most mainstream languages, type information can flow +from the leaves of the expression tree (e.g., the expression 'pi', which +refers to a double) up to the root (the type of the variable 'three'). +However, Swift also allows type information to flow from the context +(e.g., the fixed type of the variable 'eFloat') at the root of the +expression tree down to the leaves (the type of the numeric literal +2.71828). This bi-directional type inference is common in languages that +use ML-like type systems, but is not present in mainstream languages +like C++, Java, C\#, or Objective-C. + +Swift implements bi-directional type inference using a constraint-based +type checker that is reminiscent of the classical Hindley-Milner type +inference algorithm. The use of a constraint system allows a +straightforward, general presentation of language semantics that is +decoupled from the actual implementation of the solver. It is expected +that the constraints themselves will be relatively stable, while the +solver will evolve over time to improve performance and diagnostics. + +The Swift language contains a number of features not part of the +Hindley-Milner type system, including constrained polymorphic types and +function overloading, which complicate the presentation and +implementation somewhat. On the other hand, Swift limits the scope of +type inference to a single expression or statement, for purely practical +reasons: we expect that we can provide better performance and vastly +better diagnostics when the problem is limited in scope. + +Type checking proceeds in three main stages: + +Constraint Generation\_ + +: Given an input expression and (possibly) additional contextual + information, generate a set of type constraints that describes the + relationships among the types of the various subexpressions. The + generated constraints may contain unknown types, represented by type + variables, which will be determined by the solver. + +Constraint Solving\_ + +: Solve the system of constraints by assigning concrete types to each + of the type variables in the constraint system. The constraint + solver should provide the most specific solution possible among + different alternatives. + +Solution Application\_ + +: Given the input expression, the set of type constraints generated + from that expression, and the set of assignments of concrete types + to each of the type variables, produce a well-typed expression that + makes all implicit conversions (and other transformations) explicit + and resolves all unknown types and overloads. This step cannot fail. + +The following sections describe these three stages of type checking, as +well as matters of performance and diagnostics. + +Constraints +----------- + +A constraint system consists of a set of type constraints. Each type +constraint either places a requirement on a single type (e.g., it is an +integer literal type) or relates two types (e.g., one is a subtype of +the other). The types described in constraints can be any type in the +Swift type system including, e.g., builtin types, tuple types, function +types, enum/struct/class types, protocol types, and generic types. +Additionally, a type can be a type variable `T` (which are typically +numbered, `T0`, `T1`, `T2`, etc., and are introduced as needed). Type +variables can be used in place of any other type, e.g., a tuple type +`(T0, Int, (T0) -> Int)` involving the type variable `T0`. + +There are a number of different kinds of constraints used to describe +the Swift type system: + +**Equality** + +: An equality constraint requires two types to be identical. For + example, the constraint `T0 == T1` effectively ensures that `T0` and + `T1` get the same concrete type binding. There are two different + flavors of equality constraints: + + > - Exact equality constraints, or "binding", written `T0 := X` + > for some type variable `T0` and type `X`, which requires that + > `T0` be exactly identical to `X`; + > - Equality constraints, written `X == Y` for types `X` and `Y`, + > which require `X` and `Y` to have the same type, ignoring + > lvalue types in the process. For example, the constraint + > `T0 == X` would be satisfied by assigning `T0` the type `X` + > and by assigning `T0` the type `@lvalue X`. + +**Subtyping** + +: A subtype constraint requires the first type to be equivalent to or + a subtype of the second. For example, a class type `Dog` is a + subtype of a class type `Animal` if `Dog` inherits from `Animal` + either directly or indirectly. Subtyping constraints are written + `X < Y`. + +**Conversion** + +: A conversion constraint requires that the first type be convertible + to the second, which includes subtyping and equality. Additionally, + it allows a user-defined conversion function to be called. + Conversion constraints are written `X T1 ==Fn T(a)` (for fresh type + variables `T0` and `T1`) captures the rvalue-to-lvalue conversion + applied on the function (`a`) and decomposes the function type into + its argument and result types. Second, the conversion constraint + `T(b) T1` treats the + subscript as a function from the key type to the value type, + represented by fresh type variables `T0` and `T1`, respectively. The + constraint `T(b) Int { return -x } + func negate(x: Double) -> Double { return -x } + +Given that there are two definitions of `negate`, what is the type of +the declaration reference expression `negate`? If one selects the first +overload, the type is `(Int) -> Int`; for the second overload, the type +is `(Double) -> Double`. However, constraint generation needs to assign +some specific type to the expression, so that its parent expressions can +refer to that type. + +Overloading in the type checker is modeled by introducing a fresh type +variable (call it `T0`) for the type of the reference to an overloaded +declaration. Then, a disjunction constraint is introduced, in which each +term binds that type variable (via an exact equality constraint) to the +type produced by one of the overloads in the overload set. In our negate +example, the disjunction is +`T0 := (Int) -> Int or T0 := (Double) -> Double`. The constraint solver, +discussed in the later section on Constraint Solving\_, explores both +possible bindings, and the overloaded reference resolves to whichever +binding results in a solution that satisfies all constraints [^1]. + +Overloading can be introduced both by expressions that refer to sets of +overloaded declarations and by member constraints that end up resolving +to a set of overloaded declarations. One particularly interesting case +is the unresolved member reference, e.g., `.name`. As noted in the prior +section, this generates the constraint `T0.name == T1`, where `T0` is a +fresh type variable that will be bound to the enum type and `T1` is a +fresh type variable that will be bound to the type of the selected +member. The issue noted in the prior section is that this constraint +does not give the solver enough information to determine `T0` without +guesswork. However, we note that the type of a enum member actually has +a regular structure. For example, consider the `Optional` type: + + enum Optional { + case None + case Some(T) + } + +The type of `Optional.Vone` is `Optional`, while the type of +`Optional.Some` is `(T) -> Optional`. In fact, the type of a enum +element can have one of two forms: it can be `T0`, for a enum element +that has no extra data, or it can be `T2 -> T0`, where `T2` is the data +associated with the enum element. For the latter case, the actual +arguments are parsed as part of the unresolved member reference, so that +a function application constraint describes their conversion to the +input type `T2`. + +#### Polymorphic Types + +The Swift language includes generics, a system of constrained parameter +polymorphism that enables polymorphic types and functions. For example, +one can implement a `min` function as, e.g.,: + + func min(x: T, y: T) -> T { + if y < x { return y } + return x + } + +Here, `T` is a generic parameter that can be replaced with any concrete +type, so long as that type conforms to the protocol `Comparable`. The +type of `min` is (internally) written as +` (x: T, y: T) -> T`, which can be read as "for all `T`, +where `T` conforms to `Comparable`, the type of the function is +`(x: T, y: T) -> T`. Different uses of the `min` function may have +different bindings for the generic parameter`T`. + +When the constraint generator encounters a reference to a generic +function, it immediately replaces each of the generic parameters within +the function type with a fresh type variable, introduces constraints on +that type variable to match the constraints listed in the generic +function, and produces a monomorphic function type based on the +newly-generated type variables. For example, the first occurrence of the +declaration reference expression `min` would result in a type +`(x : T0, y : T0) -> T0`, where `T0` is a fresh type variable, as well +as the subtype constraint `T0 < Comparable`, which expresses protocol +conformance. The next occurrence of the declaration reference expression +`min` would produce the type `(x : T1, y : T1) -> T1`, where `T1` is a +fresh type variable (and therefore distinct from `T0`), and so on. This +replacement process is referred to as "opening" the generic function +type, and is a fairly simple (but effective) way to model the use of +polymorphic functions within the constraint system without complicating +the solver. Note that this immediate opening of generic function types +is only valid because Swift does not support first-class polymorphic +functions, e.g., one cannot declare a variable of type ` T -> T`. + +Uses of generic types are also immediately opened by the constraint +solver. For example, consider the following generic dictionary type: + + class Dictionary { + // ... + } + +When the constraint solver encounters the expression `Dictionary()`, it +opens up the type `Dictionary`---which has not been provided with any +specific generic arguments---to the type `Dictionary`, for fresh +type variables `T0` and `T1`, and introduces the constraint +`T0 conforms to Hashable`. This allows the actual key and value types of +the dictionary to be determined by the context of the expression. As +noted above for first-class polymorphic functions, this immediate +opening is valid because an unbound generic type, i.e., one that does +not have specified generic arguments, cannot be used except where the +generic arguments can be inferred. + +Constraint Solving +------------------ + +The primary purpose of the constraint solver is to take a given set of +constraints and determine the most specific type binding for each of the +type variables in the constraint system. As part of this determination, +the constraint solver also resolves overloaded declaration references by +selecting one of the overloads. + +Solving the constraint systems generated by the Swift language can, in +the worst case, require exponential time. Even the classic +Hindley-Milner type inference algorithm requires exponential time, and +the Swift type system introduces additional complications, especially +overload resolution. However, the problem size for any particular +expression is still fairly small, and the constraint solver can employ a +number of tricks to improve performance. The Performance\_ section +describes some tricks that have been implemented or are planned, and it +is expected that the solver will be extended with additional tricks +going forward. + +This section will focus on the basic ideas behind the design of the +solver, as well as the type rules that it applies. + +### Simplification + +The constraint generation process introduces a number of constraints +that can be immediately solved, either directly (because the solution is +obvious and trivial) or by breaking the constraint down into a number of +smaller constraints. This process, referred to as *simplification*, +canonicalizes a constraint system for later stages of constraint +solving. It is also re-invoked each time the constraint solver makes a +guess (at resolving an overload or binding a type variable, for +example), because each such guess often leads to other simplifications. +When all type variables and overloads have been resolved, simplification +terminates the constraint solving process either by detecting a trivial +constraint that is not satisfied (hence, this is not a proper solution) +or by reducing the set of constraints down to only simple constraints +that are trivially satisfied. + +The simplification process breaks down constraints into simpler +constraints, and each different kind of constraint is handled by +different rules based on the Swift type system. The constraints fall +into five categories: relational constraints, member constraints, type +properties, conjunctions, and disjunctions. Only the first three kinds +of constraints have interesting simplification rules, and are discussed +in the following sections. + +#### Relational Constraints + +Relational constraints describe a relationship between two types. This +category covers the equality, subtyping, and conversion constraints, and +provides the most common simplifications. The simplification of +relationship constraints proceeds by comparing the structure of the two +types and applying the typing rules of the Swift language to generate +additional constraints. For example, if the constraint is a conversion +constraint: + + A -> B D + +then both types are function types, and we can break down this +constraint into two smaller constraints `C < A` and `B < D` by applying +the conversion rule for function types. Similarly, one can destroy all +of the various type constructors---tuple types, generic type +specializations, lvalue types, etc.---to produce simpler requirements, +based on the type rules of the language [^2]. + +Relational constraints involving a type variable on one or both sides +generally cannot be solved directly. Rather, these constraints inform +the solving process later by providing possible type bindings, described +in the Type Variable Bindings\_ section. The exception is an equality +constraint between two type variables, e.g., `T0 == T1`. These +constraints are simplified by unifying the equivalence classes of `T0` +and `T1` (using a basic union-find algorithm), such that the solver need +only determine a binding for one of the type variables (and the other +gets the same binding). + +#### Member Constraints + +Member constraints specify that a certain type has a member of a given +name and provide a binding for the type of that member. A member +constraint `A.member == B` can be simplified when the type of `A` is +determined to be a nominal or tuple type, in which case name lookup can +resolve the member name to an actual declaration. That declaration has +some type `C`, so the member constraint is simplified to the exact +equality constraint`B := C`. + +The member name may refer to a set of overloaded declarations. In this +case, the type `C` is a fresh type variable (call it `T0`). A +disjunction constraint is introduced, each term of which new overload +set binds a different declaration's type to `T0`, as described in the +section on Overloading\_. + +The kind of member constraint---type or value---also affects the +declaration type `C`. A type constraint can only refer to member types, +and `C` will be the declared type of the named member. A value +constraint, on the other hand, can refer to either a type or a value, +and `C` is the type of a reference to that entity. For a reference to a +type, `C` will be a metatype of the declared type. + +### Strategies + +The basic approach to constraint solving is to simplify the constraints +until they can no longer be simplified, then produce (and check) +educated guesses about which declaration from an overload set should be +selected or what concrete type should be bound to a given type variable. +Each guess is tested as an assumption, possibly with other guesses, +until the solver either arrives at a solution or concludes that the +guess was incorrect. + +Within the implementation, each guess is modeled as an assumption within +a new solver scope. The solver scope inherits all of the constraints, +overload selections, and type variable bindings of its parent solver +scope, then adds one more guess. As such, the solution space explored by +the solver can be viewed as a tree, where the top-most node is the +constraint system generated directly from the expression. The leaves of +the tree are either solutions to the type-checking problem (where all +constraints have been simplified away) or represent sets of assumptions +that do not lead to a solution. + +The following sections describe the techniques used by the solver to +produce derived constraint systems that explore the solution space. + +#### Overload Selection + +Overload selection is the simplest way to make an assumption. For an +overload set that introduced a disjunction constraint +`T0 := A1 or T0 := A2 or ... or T0 := AN` into the constraint system, +each term in the disjunction will be visited separately. Each solver +state binds the type variable `T0` and explores whether the selected +overload leads to a suitable solution. + +#### Type Variable Bindings + +A second way in which the solver makes assumptions is to guess at the +concrete type to which a given type variable should be bound. That type +binding is then introduced in a new, derived constraint system to +determine if the binding is feasible. + +The solver does not conjure concrete type bindings from nothing, nor +does it perform an exhaustive search. Rather, it uses the constraints +placed on that type variable to produce potential candidate types. There +are several strategies employed by the solver. + +##### Meets and Joins + +A given type variable `T0` often has relational constraints placed on it +that relate it to concrete types, e.g., `T0 B` in the latter if `A` is +convertible to `B`. `B` would therefore be higher in the lattice than +`A`, and the topmost element of the lattice is the element to which all +types can be converted, `protocol<>` (often called "top"). + +The concrete types "above" and "below" a given type variable provide +bounds on the possible concrete types that can be assigned to that type +variable. The solver computes [^3] the join of the types "below" the +type variable, i.e., the most specific (lowest) type to which all of the +types "below" can be converted, and uses that join as a starting guess. + +##### Supertype Fallback + +The join of the "below" types computed as a starting point may be too +specific, due to constraints that involve the type variable but weren't +simple enough to consider as part of the join. To cope with such cases, +if no solution can be found with the join of the "below" types, the +solver creates a new set of derived constraint systems with weaker +assumptions, corresponding to each of the types that the join is +directly convertible to. For example, if the join was some class +`Derived`, the supertype fallback would then try the class `Base` from +which `Derived` directly inherits. This fallback process continues until +the types produced are no longer convertible to the meet of types +"above" the type variable, i.e., the least specific (highest) type from +which all of the types "above" the type variable can be converted [^4]. + +##### Default Literal Types + +If a type variable is bound by a conformance constraint to one of the +literal protocols, "`T0` conforms to `IntegerLiteralConvertible`", then +the constraint solver will guess that the type variable can be bound to +the default literal type for that protocol. For example, `T0` would get +the default integer literal type `Int`, allowing one to type-check +expressions with too little type information to determine the types of +these literals, e.g., `-1`. + +### Comparing Solutions + +The solver explores a potentially large solution space, and it is +possible that it will find multiple solutions to the constraint system +as given. Such cases are not necessarily ambiguities, because the solver +can then compare the solutions to to determine whether one of the +solutions is better than all of the others. To do so, it computes a +"score" for each solution based on a number of factors: + +- How many user-defined conversions were applied. +- How many non-trivial function conversions were applied. +- How many literals were given "non-default" types. + +Solutions with smaller scores are considered better solutions. When two +solutions have the same score, the type variables and overload choices +of the two systems are compared to produce a relative score: + +- If the two solutions have selected different type variable bindings + for a type variable where a "more specific" type variable is a + better match, and one of the type variable bindings is a subtype of + the other, the solution with the subtype earns +1. +- If an overload set has different selected overloads in the two + solutions, the overloads are compared. If the type of the overload + picked in one solution is a subtype of the type of the overload + picked in the other solution, then first solution earns +1. + +The solution with the greater relative score is considered to be better +than the other solution. + +Solution Application +-------------------- + +Once the solver has produced a solution to the constraint system, that +solution must be applied to the original expression to produce a fully +type-checked expression that makes all implicit conversions and resolved +overloads explicit. This application process walks the expression tree +from the leaves to the root, rewriting each expression node based on the +kind of expression: + +*Declaration references* + +: Declaration references are rewritten with the precise type of the + declaration as referenced. For overloaded declaration references, + the `Overload*Expr` node is replaced with a simple declaration + reference expression. For references to polymorphic functions or + members of generic types, a `SpecializeExpr` node is introduced to + provide substitutions for all of the generic parameters. + +*Member references* + +: References to members are similar to declaration references. + However, they have the added constraint that the base expression + needs to be a reference. Therefore, an rvalue of non-reference type + will be materialized to produce the necessary reference. + +*Literals* + +: Literals are converted to the appropriate literal type, which + typically involves introducing calls to the witnesses for the + appropriate literal protocols. + +*Closures* + +: Since the closure has acquired a complete function type, the body of + the closure is type-checked with that complete function type. + +The solution application step cannot fail for any type checking rule +modeled by the constraint system. However, there are some failures that +are intentionally left to the solution application phase, such as a +postfix '!' applied to a non-optional type. + +### Locators + +During constraint generation and solving, numerous constraints are +created, broken apart, and solved. During constraint application as well +as during diagnostics emission, it is important to track the +relationship between the constraints and the actual expressions from +which they originally came. For example, consider the following type +checking problem: + + struct X { + // user-defined conversions + func [conversion] __conversion () -> String { /* ... */ } + func [conversion] __conversion () -> Int { /* ... */ } + } + + func f(i : Int, s : String) { } + + var x : X + f(10.5, x) + +This constraint system generates the constraints "`T(f)` ==Fn +`T0 -> T1`" (for fresh variables `T0` and `T1`), "`(T2, X)` <c `T0`" +(for fresh variable `T2`) and +"`T2 conforms to`FloatLiteralConvertible`". As part of the solution, after`T0`is replaced with`(i +: Int, s : +String)`, the second of these constraints is broken down into "`T2 <c +`Int`" and "`X` <c `String`". These two constraints are interesting +for different reasons: the first will fail, because `Int` does not +conform to `FloatLiteralConvertible`. The second will succeed by +selecting one of the (overloaded) conversion functions. + +In both of these cases, we need to map the actual constraint of interest +back to the expressions they refer to. In the first case, we want to +report not only that the failure occurred because `Int` is not +`FloatLiteralConvertible`, but we also want to point out where the `Int` +type actually came from, i.e., in the parameter. In the second case, we +want to determine which of the overloaded conversion functions was +selected to perform the conversion, so that conversion function can be +called by constraint application if all else succeeds. + +*Locators* address both issues by tracking the location and derivation +of constraints. Each locator is anchored at a specific expression, i.e., +the function application `f(10.5, x)`, and contains a path of zero or +more derivation steps from that anchor. For example, the "`T(f)` ==Fn +`T0 -> T1`" constraint has a locator that is anchored at the function +application and a path with the "apply function" derivation step, +meaning that this is the function being applied. Similarly, the +"`(T2, X)` <c `T0` constraint has a locator anchored at the function +application and a path with the "apply argument" derivation step, +meaning that this is the argument to the function. + +When constraints are simplified, the resulting constraints have locators +with longer paths. For example, when a conversion constraint between two +tuples is simplified conversion constraints between the corresponding +tuple elements, the resulting locators refer to specific elements. For +example, the `T2 apply argument -> tuple element #1 -> conversion member + +When the solver selects a particular overload from the overload set, it +records the selected overload based on the locator of the overload set. +When it comes time to perform constraint application, the locator is +recreated based on context (as the bottom-up traversal walks the +expressions to rewrite them with their final types) and used to find the +appropriate conversion to call. The same mechanism is used to select the +appropriate overload when an expression refers directly to an overloaded +function. Additionally, when comparing two solutions to the same +constraint system, overload sets present in both solutions can be found +by comparing the locators for each of the overload choices made in each +solution. Naturally, all of these operations require locators to be +unique, which occurs in the constraint system itself. + +#### Simplifying Locators + +Locators provide the derivation of location information that follows the +path of the solver, and can be used to query and recover the important +decisions made by the solver. However, the locators determined by the +solver may not directly refer to the most specific expression for the +purposes of identifying the corresponding source location. For example, +the failed constraint "`Int` conforms to `FloatLiteralConvertible`" can +most specifically by centered on the floating-point literal `10.5`, but +its locator is: + + function application -> apply argument -> tuple element #0 + +The process of locator simplification maps a locator to its most +specific expression. Essentially, it starts at the anchor of the locator +(in this case, the application `f(10.5, x)`) and then walks the path, +matching derivation steps to subexpressions. The "function application" +derivation step extracts the argument (`(10.5, x)`). Then, the "tuple +element \#0" derivation extracts the tuple element 0 subexpression, +`10.5`, at which point we have traversed the entire path and now have +the most specific expression for source-location purposes. + +Simplification does not always exhaust the complete path. For example, +consider a slight modification to our example, so that the argument to +`f` is provided by another call, we get a different result entirely: + + func f(i : Int, s : String) { } + func g() -> (f : Float, x : X) { } + + f(g()) + +Here, the failing constraint is `Float apply argument -> tuple element #0 + +When we simplify this locator, we start with `f(g())`. The "apply +argument" derivation step takes us to the argument expression `g()`. +Here, however, there is no subexpression for the first tuple element of +`g()`, because it's simple part of the tuple returned from `g`. At this +point, simplification ceases, and creates the simplified locator: + + function application of g -> tuple element #0 + +Performance +----------- + +The performance of the type checker is dependent on a number of factors, +but the chief concerns are the size of the solution space (which is +exponential in the worst case) and the effectiveness of the solver in +exploring that solution space. This section describes some of the +techniques used to improve solver performance, many of which can +doubtless be improved. + +### Constraint Graph + +The constraint graph describes the relationships among type variables in +the constraint system. Each vertex in the constraint graph corresponds +to a single type variable. The edges of the graph correspond to +constraints in the constraint system, relating sets of type variables +together. Technically, this makes the constraint graph a *multigraph*, +although the internal representation is more akin to a graph with +multiple kinds of edges: each vertex (node) tracks the set of +constraints that mention the given type variable as well as the set of +type variables that are adjacent to this type variable. A vertex also +includes information about the equivalence class corresponding to a +given type variable (when type variables have been merged) or the +binding of a type variable to a specific type. + +The constraint graph is critical to a number of solver optimizations. +For example, it is used to compute the connected components within the +constraint graph, so that each connected component can be solved +independently. The partial results from all of the connected components +are then combined into a complete solution. Additionally, the constraint +graph is used to direct simplification, described below. + +### Simplification Worklist + +When the solver has attempted a type variable binding, that binding +often leads to additional simplifications in the constraint system. The +solver will query the constraint graph to determine which constraints +mention the type variable and will place those constraints onto the +simplification worklist. If those constraints can be simplified further, +it may lead to additional type variable bindings, which in turn adds +more constraints to the worklist. Once the worklist is exhausted, +simplification has completed. The use of the worklist eliminates the +need to reprocess constraints that could not have changed because the +type variables they mention have not changed. + +### Solver Scopes + +The solver proceeds through the solution space in a depth-first manner. +Whenever the solver is about to make a guess---such as a speculative +type variable binding or the selection of a term from a disjunction---it +introduces a new solver scope to capture the results of that assumption. +Subsequent solver scopes are nested as the solver builds up a set of +assumptions, eventually leading to either a solution or an error. When a +solution is found, the stack of solver scopes contains all of the +assumptions needed to produce that solution, and is saved in a separate +solution data structure. + +The solver scopes themselves are designed to be fairly cheap to create +and destroy. To support this, all of the major data structures used by +the constraint solver have reversible operations, allowing the solver to +easily backtrack. For example, the addition of a constraint to the +constraint graph can be reversed by removing that same constraint. The +constraint graph tracks all such additions in a stack: pushing a new +solver scope stores a marker to the current top of the stack, and +popping that solver scope reverses all of the operations on that stack +until it hits the marker. + +### Online Scoring + +As the solver evaluates potential solutions, it keeps track of the score +of the current solution and of the best complete solution found thus +far. If the score of the current solution is ever greater than that of +the best complete solution, it abandons the current solution and +backtracks to continue its search. + +The solver makes some attempt at evaluating cheaper solutions before +more expensive solutions. For example, it will prefer to try normal +conversions before user-defined conversions, prefer the "default" +literal types over other literal types, and prefer cheaper conversions +to more expensive conversions. However, some of the rules are fairly ad +hoc, and could benefit from more study. + +### Arena Memory Management + +Each constraint system introduces its own memory allocation arena, +making allocations cheap and deallocation essentially free. The +allocation arena extends all the way into the AST context, so that types +composed of type variables (e.g., `T0 -> T1`) will be allocated within +the constraint system's arena rather than the permanent arena. Most data +structures involved in constraint solving use this same arena. + +Diagnostics +----------- + +The diagnostics produced by the type checker are currently terrible. We +plan to do something about this, eventually. We also believe that we can +implement some heroics, such as spell-checking that takes into account +the surrounding expression to only provide well-typed suggestions. + +[^1]: It is possible that both overloads will result in a solution, in + which case the solutions will be ranked based on the rules discussed + in the section Comparing Solutions\_. + +[^2]: As of the time of this writing, the type rules of Swift have not + specifically been documented outside of the source code. The + constraints-based type checker contains a function `matchTypes` that + documents and implements each of these rules. A future revision of + this document will provide a more readily-accessible version. + +[^3]: More accurately, as of this writing, "will compute". The solver + doesn't current compute meets and joins properly. Rather, it + arbitrarily picks one of the constraints "below" to start with. + +[^4]: Again, as of this writing, the solver doesn't actually compute + meets and joins, so the solver continues until it runs out of + supertypes to enumerate. diff --git a/docs/TypeChecker.rst b/docs/TypeChecker.rst deleted file mode 100644 index 400c799bfc6ce..0000000000000 --- a/docs/TypeChecker.rst +++ /dev/null @@ -1,975 +0,0 @@ -.. @raise litre.TestsAreMissing - -Type Checker Design and Implementation -======================================== - -Purpose ------------------ - -This document describes the design and implementation of the Swift -type checker. It is intended for developers who wish to modify, -extend, or improve on the type checker, or simply to understand in -greater depth how the Swift type system works. Familiarity with the -Swift programming language is assumed. - -Approach -------------------- - -The Swift language and its type system incorporate a number of popular -language features, including object-oriented programming via classes, -function and operator overloading, subtyping, and constrained -parametric polymorphism. Swift makes extensive use of type inference, -allowing one to omit the types of many variables and expressions. For -example:: - - func round(x: Double) -> Int { /* ... */ } - var pi: Double = 3.14159 - var three = round(pi) // 'three' has type 'Int' - - func identity(x: T) -> T { return x } - var eFloat: Float = -identity(2.71828) // numeric literal gets type 'Float' - -Swift's type inference allows type information to flow in two -directions. As in most mainstream languages, type information can flow -from the leaves of the expression tree (e.g., the expression 'pi', -which refers to a double) up to the root (the type of the variable -'three'). However, Swift also allows type information to flow from the -context (e.g., the fixed type of the variable 'eFloat') at the root of -the expression tree down to the leaves (the type of the numeric -literal 2.71828). This bi-directional type inference is common in -languages that use ML-like type systems, but is not present in -mainstream languages like C++, Java, C#, or Objective-C. - -Swift implements bi-directional type inference using a -constraint-based type checker that is reminiscent of the classical -Hindley-Milner type inference algorithm. The use of a constraint -system allows a straightforward, general presentation of language -semantics that is decoupled from the actual implementation of the -solver. It is expected that the constraints themselves will be -relatively stable, while the solver will evolve over time to improve -performance and diagnostics. - -The Swift language contains a number of features not part of the -Hindley-Milner type system, including constrained polymorphic types -and function overloading, which complicate the presentation and -implementation somewhat. On the other hand, Swift limits the scope of -type inference to a single expression or statement, for purely -practical reasons: we expect that we can provide better performance -and vastly better diagnostics when the problem is limited in scope. - -Type checking proceeds in three main stages: - -`Constraint Generation`_ - Given an input expression and (possibly) additional contextual - information, generate a set of type constraints that describes the - relationships among the types of the various subexpressions. The - generated constraints may contain unknown types, represented by type - variables, which will be determined by the solver. - -`Constraint Solving`_ - Solve the system of constraints by assigning concrete types to each - of the type variables in the constraint system. The constraint - solver should provide the most specific solution possible among - different alternatives. - -`Solution Application`_ - Given the input expression, the set of type constraints generated - from that expression, and the set of assignments of concrete types - to each of the type variables, produce a well-typed expression that - makes all implicit conversions (and other transformations) explicit - and resolves all unknown types and overloads. This step cannot fail. - -The following sections describe these three stages of type checking, -as well as matters of performance and diagnostics. - -Constraints ----------------- -A constraint system consists of a set of type constraints. Each type -constraint either places a requirement on a single type (e.g., it is -an integer literal type) or relates two types (e.g., one is a subtype -of the other). The types described in constraints can be any type in -the Swift type system including, e.g., builtin types, tuple types, -function types, enum/struct/class types, protocol types, and generic -types. Additionally, a type can be a type variable ``T`` (which are -typically numbered, ``T0``, ``T1``, ``T2``, etc., and are introduced -as needed). Type variables can be used in place of any other type, -e.g., a tuple type ``(T0, Int, (T0) -> Int)`` involving the type -variable ``T0``. - -There are a number of different kinds of constraints used to describe -the Swift type system: - -**Equality** - An equality constraint requires two types to be identical. For - example, the constraint ``T0 == T1`` effectively ensures that ``T0`` and - ``T1`` get the same concrete type binding. There are two different - flavors of equality constraints: - - - Exact equality constraints, or "binding", written ``T0 := X`` - for some type variable ``T0`` and type ``X``, which requires - that ``T0`` be exactly identical to ``X``; - - Equality constraints, written ``X == Y`` for types ``X`` and - ``Y``, which require ``X`` and ``Y`` to have the same type, - ignoring lvalue types in the process. For example, the - constraint ``T0 == X`` would be satisfied by assigning ``T0`` - the type ``X`` and by assigning ``T0`` the type ``@lvalue X``. - -**Subtyping** - A subtype constraint requires the first type to be equivalent to or - a subtype of the second. For example, a class type ``Dog`` is a - subtype of a class type ``Animal`` if ``Dog`` inherits from - ``Animal`` either directly or indirectly. Subtyping constraints are - written ``X < Y``. - -**Conversion** - A conversion constraint requires that the first type be convertible - to the second, which includes subtyping and equality. Additionally, - it allows a user-defined conversion function to be - called. Conversion constraints are written ``X T1 ==Fn T(a)`` (for fresh - type variables ``T0`` and ``T1``) captures the rvalue-to-lvalue - conversion applied on the function (``a``) and decomposes the - function type into its argument and result types. Second, the - conversion constraint ``T(b) T1`` treats the - subscript as a function from the key type to the value type, - represented by fresh type variables ``T0`` and ``T1``, - respectively. The constraint ``T(b) Int { return -x } - func negate(x: Double) -> Double { return -x } - -Given that there are two definitions of ``negate``, what is the type -of the declaration reference expression ``negate``? If one selects the -first overload, the type is ``(Int) -> Int``; for the second overload, -the type is ``(Double) -> Double``. However, constraint generation -needs to assign some specific type to the expression, so that its -parent expressions can refer to that type. - -Overloading in the type checker is modeled by introducing a fresh type -variable (call it ``T0``) for the type of the reference to an -overloaded declaration. Then, a disjunction constraint is introduced, -in which each term binds that type variable (via an exact equality -constraint) to the type produced by one of the overloads in the -overload set. In our negate example, the disjunction is -``T0 := (Int) -> Int or T0 := (Double) -> Double``. The constraint -solver, discussed in the later section on `Constraint Solving`_, -explores both possible bindings, and the overloaded reference resolves -to whichever binding results in a solution that satisfies all -constraints [#]_. - -Overloading can be introduced both by expressions that refer to sets -of overloaded declarations and by member constraints that end up -resolving to a set of overloaded declarations. One particularly -interesting case is the unresolved member reference, e.g., -``.name``. As noted in the prior section, this generates the -constraint ``T0.name == T1``, where ``T0`` is a fresh type variable -that will be bound to the enum type and ``T1`` is a fresh type -variable that will be bound to the type of the selected member. The -issue noted in the prior section is that this constraint does not give -the solver enough information to determine ``T0`` without -guesswork. However, we note that the type of a enum member actually -has a regular structure. For example, consider the ``Optional`` type:: - - enum Optional { - case None - case Some(T) - } - -The type of ``Optional.Vone`` is ``Optional``, while the type of -``Optional.Some`` is ``(T) -> Optional``. In fact, the -type of a enum element can have one of two forms: it can be ``T0``, -for a enum element that has no extra data, or it can be ``T2 -> T0``, -where ``T2`` is the data associated with the enum element. For the -latter case, the actual arguments are parsed as part of the unresolved -member reference, so that a function application constraint describes -their conversion to the input type ``T2``. - -Polymorphic Types -'''''''''''''''''''''''''''''''''''''''''''''' - -The Swift language includes generics, a system of constrained -parameter polymorphism that enables polymorphic types and -functions. For example, one can implement a ``min`` function as, -e.g.,:: - - func min(x: T, y: T) -> T { - if y < x { return y } - return x - } - -Here, ``T`` is a generic parameter that can be replaced with any -concrete type, so long as that type conforms to the protocol -``Comparable``. The type of ``min`` is (internally) written as `` (x: T, y: T) -> T``, which can be read as "for all ``T``, -where ``T`` conforms to ``Comparable``, the type of the function is -``(x: T, y: T) -> T``. Different uses of the ``min`` function may -have different bindings for the generic parameter``T``. - -When the constraint generator encounters a reference to a generic -function, it immediately replaces each of the generic parameters within -the function type with a fresh type variable, introduces constraints -on that type variable to match the constraints listed in the generic -function, and produces a monomorphic function type based on the -newly-generated type variables. For example, the first occurrence of -the declaration reference expression ``min`` would result in a type -``(x : T0, y : T0) -> T0``, where ``T0`` is a fresh type variable, as -well as the subtype constraint ``T0 < Comparable``, which expresses -protocol conformance. The next occurrence of the declaration reference -expression ``min`` would produce the type ``(x : T1, y : T1) -> T1``, -where ``T1`` is a fresh type variable (and therefore distinct from -``T0``), and so on. This replacement process is referred to as -"opening" the generic function type, and is a fairly simple (but -effective) way to model the use of polymorphic functions within the -constraint system without complicating the solver. Note that this -immediate opening of generic function types is only valid because -Swift does not support first-class polymorphic functions, e.g., one -cannot declare a variable of type `` T -> T``. - -Uses of generic types are also immediately opened by the constraint -solver. For example, consider the following generic dictionary type:: - - class Dictionary { - // ... - } - -When the constraint solver encounters the expression `` -Dictionary()``, it opens up the type ``Dictionary``---which has not -been provided with any specific generic arguments---to the type -``Dictionary``, for fresh type variables ``T0`` and ``T1``, -and introduces the constraint ``T0 conforms to Hashable``. This allows -the actual key and value types of the dictionary to be determined by -the context of the expression. As noted above for first-class -polymorphic functions, this immediate opening is valid because an -unbound generic type, i.e., one that does not have specified generic -arguments, cannot be used except where the generic arguments can be -inferred. - -Constraint Solving ------------------------------ -The primary purpose of the constraint solver is to take a given set of -constraints and determine the most specific type binding for each of the type -variables in the constraint system. As part of this determination, the -constraint solver also resolves overloaded declaration references by -selecting one of the overloads. - -Solving the constraint systems generated by the Swift language can, in -the worst case, require exponential time. Even the classic -Hindley-Milner type inference algorithm requires exponential time, and -the Swift type system introduces additional complications, especially -overload resolution. However, the problem size for any particular -expression is still fairly small, and the constraint solver can employ -a number of tricks to improve performance. The Performance_ section -describes some tricks that have been implemented or are planned, and -it is expected that the solver will be extended with additional tricks -going forward. - -This section will focus on the basic ideas behind the design of the -solver, as well as the type rules that it applies. - -Simplification -``````````````````` -The constraint generation process introduces a number of constraints -that can be immediately solved, either directly (because the solution -is obvious and trivial) or by breaking the constraint down into a -number of smaller constraints. This process, referred to as -*simplification*, canonicalizes a constraint system for later stages -of constraint solving. It is also re-invoked each time the constraint -solver makes a guess (at resolving an overload or binding a type -variable, for example), because each such guess often leads to other -simplifications. When all type variables and overloads have been -resolved, simplification terminates the constraint solving process -either by detecting a trivial constraint that is not satisfied (hence, -this is not a proper solution) or by reducing the set of constraints -down to only simple constraints that are trivially satisfied. - -The simplification process breaks down constraints into simpler -constraints, and each different kind of constraint is handled by -different rules based on the Swift type system. The constraints fall -into five categories: relational constraints, member constraints, -type properties, conjunctions, and disjunctions. Only the first three -kinds of constraints have interesting simplification rules, and are -discussed in the following sections. - -Relational Constraints -'''''''''''''''''''''''''''''''''''''''''''''''' - -Relational constraints describe a relationship between two types. This -category covers the equality, subtyping, and conversion constraints, -and provides the most common simplifications. The simplification of -relationship constraints proceeds by comparing the structure of the -two types and applying the typing rules of the Swift language to -generate additional constraints. For example, if the constraint is a -conversion constraint:: - - A -> B D - -then both types are function types, and we can break down this -constraint into two smaller constraints ``C < A`` and ``B < D`` by -applying the conversion rule for function types. Similarly, one can -destroy all of the various type constructors---tuple types, generic -type specializations, lvalue types, etc.---to produce simpler -requirements, based on the type rules of the language [#]_. - -Relational constraints involving a type variable on one or both sides -generally cannot be solved directly. Rather, these constraints inform -the solving process later by providing possible type bindings, -described in the `Type Variable Bindings`_ section. The exception is -an equality constraint between two type variables, e.g., ``T0 == -T1``. These constraints are simplified by unifying the equivalence -classes of ``T0`` and ``T1`` (using a basic union-find algorithm), -such that the solver need only determine a binding for one of the type -variables (and the other gets the same binding). - -Member Constraints -''''''''''''''''''''''''''''''''''''''''''' - -Member constraints specify that a certain type has a member of a given -name and provide a binding for the type of that member. A member -constraint ``A.member == B`` can be simplified when the type of ``A`` -is determined to be a nominal or tuple type, in which case name lookup -can resolve the member name to an actual declaration. That declaration -has some type ``C``, so the member constraint is simplified to the -exact equality constraint``B := C``. - -The member name may refer to a set of overloaded declarations. In this -case, the type ``C`` is a fresh type variable (call it ``T0``). A -disjunction constraint is introduced, each term of which new overload -set binds a different declaration's type to ``T0``, as described in -the section on Overloading_. - -The kind of member constraint---type or value---also affects the -declaration type ``C``. A type constraint can only refer to member -types, and ``C`` will be the declared type of the named member. A -value constraint, on the other hand, can refer to either a type or a -value, and ``C`` is the type of a reference to that entity. For a -reference to a type, ``C`` will be a metatype of the declared type. - - -Strategies -``````````````````````````````` -The basic approach to constraint solving is to simplify the -constraints until they can no longer be simplified, then produce (and -check) educated guesses about which declaration from an overload set -should be selected or what concrete type should be bound to a given -type variable. Each guess is tested as an assumption, possibly with -other guesses, until the solver either arrives at a solution or -concludes that the guess was incorrect. - -Within the implementation, each guess is modeled as an assumption -within a new solver scope. The solver scope inherits all of the -constraints, overload selections, and type variable bindings of its -parent solver scope, then adds one more guess. As such, the solution -space explored by the solver can be viewed as a tree, where the -top-most node is the constraint system generated directly from the -expression. The leaves of the tree are either solutions to the -type-checking problem (where all constraints have been simplified -away) or represent sets of assumptions that do not lead to a solution. - -The following sections describe the techniques used by the solver to -produce derived constraint systems that explore the solution space. - -Overload Selection -''''''''''''''''''''''''''''''''''''''''''''''''''''' -Overload selection is the simplest way to make an assumption. For an -overload set that introduced a disjunction constraint -``T0 := A1 or T0 := A2 or ... or T0 := AN`` into the constraint -system, each term in the disjunction will be visited separately. Each -solver state binds the type variable ``T0`` and explores -whether the selected overload leads to a suitable solution. - -Type Variable Bindings -''''''''''''''''''''''''''''''''''''''''''''''''''''' -A second way in which the solver makes assumptions is to guess at the -concrete type to which a given type variable should be bound. That -type binding is then introduced in a new, derived constraint system to -determine if the binding is feasible. - -The solver does not conjure concrete type bindings from nothing, nor -does it perform an exhaustive search. Rather, it uses the constraints -placed on that type variable to produce potential candidate -types. There are several strategies employed by the solver. - -Meets and Joins -.......................................... -A given type variable ``T0`` often has relational constraints -placed on it that relate it to concrete types, e.g., ``T0 B`` in the latter if ``A`` is convertible to ``B``. ``B`` would -therefore be higher in the lattice than ``A``, and the topmost element -of the lattice is the element to which all types can be converted, -``protocol<>`` (often called "top"). - -The concrete types "above" and "below" a given type variable provide -bounds on the possible concrete types that can be assigned to that -type variable. The solver computes [#]_ the join of the types "below" -the type variable, i.e., the most specific (lowest) type to which all -of the types "below" can be converted, and uses that join as a -starting guess. - - -Supertype Fallback -.......................................... -The join of the "below" types computed as a starting point may be too -specific, due to constraints that involve the type variable but -weren't simple enough to consider as part of the join. To cope with -such cases, if no solution can be found with the join of the "below" -types, the solver creates a new set of derived constraint systems with -weaker assumptions, corresponding to each of the types that the join -is directly convertible to. For example, if the join was some class -``Derived``, the supertype fallback would then try the class ``Base`` -from which ``Derived`` directly inherits. This fallback process -continues until the types produced are no longer convertible to the -meet of types "above" the type variable, i.e., the least specific -(highest) type from which all of the types "above" the type variable -can be converted [#]_. - - -Default Literal Types -.......................................... -If a type variable is bound by a conformance constraint to one of the -literal protocols, "``T0`` conforms to ``IntegerLiteralConvertible``", -then the constraint solver will guess that the type variable can be -bound to the default literal type for that protocol. For example, -``T0`` would get the default integer literal type ``Int``, allowing -one to type-check expressions with too little type information to -determine the types of these literals, e.g., ``-1``. - -Comparing Solutions -````````````````````````` -The solver explores a potentially large solution space, and it is -possible that it will find multiple solutions to the constraint system -as given. Such cases are not necessarily ambiguities, because the -solver can then compare the solutions to to determine whether one of -the solutions is better than all of the others. To do so, it computes -a "score" for each solution based on a number of factors: - -- How many user-defined conversions were applied. -- How many non-trivial function conversions were applied. -- How many literals were given "non-default" types. - -Solutions with smaller scores are considered better solutions. When -two solutions have the same score, the type variables and overload -choices of the two systems are compared to produce a relative score: - -- If the two solutions have selected different type variable bindings - for a type variable where a "more specific" type variable is a - better match, and one of the type variable bindings is a subtype of - the other, the solution with the subtype earns +1. - -- If an overload set has different selected overloads in the two - solutions, the overloads are compared. If the type of the - overload picked in one solution is a subtype of the type of - the overload picked in the other solution, then first solution earns - +1. - -The solution with the greater relative score is considered to be -better than the other solution. - -Solution Application -------------------------- -Once the solver has produced a solution to the constraint system, that -solution must be applied to the original expression to produce a fully -type-checked expression that makes all implicit conversions and -resolved overloads explicit. This application process walks the -expression tree from the leaves to the root, rewriting each expression -node based on the kind of expression: - -*Declaration references* - Declaration references are rewritten with the precise type of the - declaration as referenced. For overloaded declaration references, the - ``Overload*Expr`` node is replaced with a simple declaration - reference expression. For references to polymorphic functions or - members of generic types, a ``SpecializeExpr`` node is introduced to - provide substitutions for all of the generic parameters. - -*Member references* - References to members are similar to declaration - references. However, they have the added constraint that the base - expression needs to be a reference. Therefore, an rvalue of - non-reference type will be materialized to produce the necessary - reference. - -*Literals* - Literals are converted to the appropriate literal type, which - typically involves introducing calls to the witnesses for the - appropriate literal protocols. - -*Closures* - Since the closure has acquired a complete function type, - the body of the closure is type-checked with that - complete function type. - -The solution application step cannot fail for any type checking rule -modeled by the constraint system. However, there are some failures -that are intentionally left to the solution application phase, such as -a postfix '!' applied to a non-optional type. - -Locators -``````````` -During constraint generation and solving, numerous constraints are -created, broken apart, and solved. During constraint application as -well as during diagnostics emission, it is important to track the -relationship between the constraints and the actual expressions from -which they originally came. For example, consider the following type -checking problem:: - - struct X { - // user-defined conversions - func [conversion] __conversion () -> String { /* ... */ } - func [conversion] __conversion () -> Int { /* ... */ } - } - - func f(i : Int, s : String) { } - - var x : X - f(10.5, x) - -This constraint system generates the constraints "``T(f)`` ==Fn ``T0 --> T1``" (for fresh variables ``T0`` and ``T1``), "``(T2, X)`` T1``" constraint has a locator that is -anchored at the function application and a path with the "apply -function" derivation step, meaning that this is the function being -applied. Similarly, the "``(T2, X)`` apply argument -> tuple element #1 -> conversion member - -When the solver selects a particular overload from the overload set, -it records the selected overload based on the locator of the overload -set. When it comes time to perform constraint application, the locator -is recreated based on context (as the bottom-up traversal walks the -expressions to rewrite them with their final types) and used to find -the appropriate conversion to call. The same mechanism is used to -select the appropriate overload when an expression refers directly to -an overloaded function. Additionally, when comparing two solutions to -the same constraint system, overload sets present in both solutions -can be found by comparing the locators for each of the overload -choices made in each solution. Naturally, all of these operations -require locators to be unique, which occurs in the constraint system -itself. - -Simplifying Locators -''''''''''''''''''''''''''''' -Locators provide the derivation of location information that follows -the path of the solver, and can be used to query and recover the -important decisions made by the solver. However, the locators -determined by the solver may not directly refer to the most specific -expression for the purposes of identifying the corresponding source -location. For example, the failed constraint "``Int`` conforms to -``FloatLiteralConvertible``" can most specifically by centered on the -floating-point literal ``10.5``, but its locator is:: - - function application -> apply argument -> tuple element #0 - -The process of locator simplification maps a locator to its most -specific expression. Essentially, it starts at the anchor of the -locator (in this case, the application ``f(10.5, x)``) and then walks -the path, matching derivation steps to subexpressions. The "function -application" derivation step extracts the argument (``(10.5, -x)``). Then, the "tuple element #0" derivation extracts the tuple -element 0 subexpression, ``10.5``, at which point we have traversed -the entire path and now have the most specific expression for -source-location purposes. - -Simplification does not always exhaust the complete path. For example, -consider a slight modification to our example, so that the argument to -``f`` is provided by another call, we get a different result -entirely:: - - func f(i : Int, s : String) { } - func g() -> (f : Float, x : X) { } - - f(g()) - -Here, the failing constraint is ``Float apply argument -> tuple element #0 - -When we simplify this locator, we start with ``f(g())``. The "apply -argument" derivation step takes us to the argument expression -``g()``. Here, however, there is no subexpression for the first tuple -element of ``g()``, because it's simple part of the tuple returned -from ``g``. At this point, simplification ceases, and creates the -simplified locator:: - - function application of g -> tuple element #0 - -Performance ------------------ -The performance of the type checker is dependent on a number of -factors, but the chief concerns are the size of the solution space -(which is exponential in the worst case) and the effectiveness of the -solver in exploring that solution space. This section describes some -of the techniques used to improve solver performance, many of which -can doubtless be improved. - -Constraint Graph -```````````````` -The constraint graph describes the relationships among type variables -in the constraint system. Each vertex in the constraint graph -corresponds to a single type variable. The edges of the graph -correspond to constraints in the constraint system, relating sets of -type variables together. Technically, this makes the constraint graph -a *multigraph*, although the internal representation is more akin to a -graph with multiple kinds of edges: each vertex (node) tracks the set -of constraints that mention the given type variable as well as the set -of type variables that are adjacent to this type variable. A vertex -also includes information about the equivalence class corresponding to -a given type variable (when type variables have been merged) or the -binding of a type variable to a specific type. - -The constraint graph is critical to a number of solver -optimizations. For example, it is used to compute the connected -components within the constraint graph, so that each connected -component can be solved independently. The partial results from all of -the connected components are then combined into a complete -solution. Additionally, the constraint graph is used to direct -simplification, described below. - -Simplification Worklist -``````````````````````` -When the solver has attempted a type variable binding, that binding -often leads to additional simplifications in the constraint -system. The solver will query the constraint graph to determine which -constraints mention the type variable and will place those constraints -onto the simplification worklist. If those constraints can be -simplified further, it may lead to additional type variable bindings, -which in turn adds more constraints to the worklist. Once the worklist -is exhausted, simplification has completed. The use of the worklist -eliminates the need to reprocess constraints that could not have -changed because the type variables they mention have not changed. - -Solver Scopes -````````````` -The solver proceeds through the solution space in a depth-first -manner. Whenever the solver is about to make a guess---such as a -speculative type variable binding or the selection of a term from a -disjunction---it introduces a new solver scope to capture the results -of that assumption. Subsequent solver scopes are nested as the solver -builds up a set of assumptions, eventually leading to either a -solution or an error. When a solution is found, the stack of solver -scopes contains all of the assumptions needed to produce that -solution, and is saved in a separate solution data structure. - -The solver scopes themselves are designed to be fairly cheap to create -and destroy. To support this, all of the major data structures used by -the constraint solver have reversible operations, allowing the solver -to easily backtrack. For example, the addition of a constraint to the -constraint graph can be reversed by removing that same constraint. The -constraint graph tracks all such additions in a stack: pushing a new -solver scope stores a marker to the current top of the stack, and -popping that solver scope reverses all of the operations on that stack -until it hits the marker. - -Online Scoring -`````````````` -As the solver evaluates potential solutions, it keeps track of the -score of the current solution and of the best complete solution found -thus far. If the score of the current solution is ever greater than -that of the best complete solution, it abandons the current solution -and backtracks to continue its search. - -The solver makes some attempt at evaluating cheaper solutions before -more expensive solutions. For example, it will prefer to try normal -conversions before user-defined conversions, prefer the "default" -literal types over other literal types, and prefer cheaper conversions -to more expensive conversions. However, some of the rules are fairly -ad hoc, and could benefit from more study. - -Arena Memory Management -``````````````````````` -Each constraint system introduces its own memory allocation arena, -making allocations cheap and deallocation essentially free. The -allocation arena extends all the way into the AST context, so that -types composed of type variables (e.g., ``T0 -> T1``) will be -allocated within the constraint system's arena rather than the -permanent arena. Most data structures involved in constraint solving -use this same arena. - -Diagnostics ------------------ -The diagnostics produced by the type checker are currently -terrible. We plan to do something about this, eventually. We also -believe that we can implement some heroics, such as spell-checking -that takes into account the surrounding expression to only provide -well-typed suggestions. - -.. [#] It is possible that both overloads will result in a solution, - in which case the solutions will be ranked based on the rules - discussed in the section `Comparing Solutions`_. - -.. [#] As of the time of this writing, the type rules of Swift have - not specifically been documented outside of the source code. The - constraints-based type checker contains a function ``matchTypes`` - that documents and implements each of these rules. A future revision - of this document will provide a more readily-accessible version. - -.. [#] More accurately, as of this writing, "will compute". The solver - doesn't current compute meets and joins properly. Rather, it - arbitrarily picks one of the constraints "below" to start with. - -.. [#] Again, as of this writing, the solver doesn't actually compute - meets and joins, so the solver continues until it runs out of - supertypes to enumerate. - diff --git a/docs/archive/LangRefNew.md b/docs/archive/LangRefNew.md new file mode 100644 index 0000000000000..21def1bd3a674 --- /dev/null +++ b/docs/archive/LangRefNew.md @@ -0,0 +1,1398 @@ +Swift Language Reference Manual +=============================== + +Introduction +------------ + +> **warning** +> +> This document has not been kept up to date. + +> **Commentary** +> +> In addition to the main spec, there are lots of open ended questions, +> justification, and ideas of what best practices should be. That random +> discussion is placed in boxes like this one to clarify what is +> normative and what is discussion. + +This is the language reference manual for the Swift language, which is +highly volatile and constantly under development. As the prototype +evolves, this document should be kept up to date with what is actually +implemented. + +The grammar and structure of the language is defined in BNF form in +yellow boxes. Examples are shown in gray boxes, and assume that the +standard library is in use (unless otherwise specified). + +### Basic Goals + +> **Commentary** +> +> A non-goal of the Swift project in general is to become some amazing +> research project. We really want to focus on delivering a real +> product, and having the design and spec co-evolve. + +In no particular order, and not explained well: + +- Support building great frameworks and applications, with a specific + focus on permiting rich and powerful APIs. +- Get the defaults right: this reduces the barrier to entry and + increases the odds that the right thing happens. +- Through our support for building great APIs, we aim to provide an + expressive and productive language that is fun to program in. +- Support low-level system programming. We should want to write + compilers, operating system kernels, and media codecs in Swift. This + means that being able to obtain high performance is really + quite important. +- Provide really great tools, like an IDE, debugger, profiling, etc. +- Where possible, steal great ideas instead of innovating new things + that will work out in unpredictable ways. It turns out that there + are a lot of good ideas already out there. +- Memory safe by default: array overrun errors, uninitialized values, + and other problems endemic to C should not occur in Swift, even if + it means some amount of runtime overhead. Eventually these checks + will be disablable for people who want ultimate performance in + production builds. +- Efficiently implementable with a static compiler: runtime + compilation is great technology and Swift may eventually get a + runtime optimizer, but it is a strong goal to be able to implement + swift with just a static compiler. +- Interoperate as transparently as possible with C, Objective-C, and + C++ without having to write an equivalent of "extern C" for every + referenced definition. +- Great support for efficient by-value types. +- Elegant and natural syntax, aiming to be familiar and easy to + transition to for "C" people. Differences from the C family should + only be done when it provides a significant win (e.g. eliminate + declarator syntax). +- Lots of other stuff too. + +A smaller wishlist goal is to support embedded sub-languages in swift, +so that we don't get the +OpenCL-is-like-C-but-very-different-in-many-details problem. + +### Basic Approach + +> **Commentary** +> +> Pushing as much of the language as realistic out of the compiler and +> into the library is generally good for a few reasons: 1) we end up +> with a smaller core language. 2) we force the language that is left to +> be highly expressive and extensible. 3) this highly expressive +> language core can then be used to build a lot of other great +> libraries, hopefully many we can't even anticipate at this point. + +The basic approach in designing and implementing the Swift prototype was +to start at the very bottom of the stack (simple expressions and the +trivial bits of the type system) and incrementally build things up one +brick at a time. There is a big focus on making things as simple as +possible and having a clean internal core. Where it makes sense, sugar +is added on top to make the core more expressive for common situations. + +One major aspect that dovetails with expressivity, learnability, and +focus on API development is that much of the language is implemented in +a standard +library <langref.stdlib> (inspired in part by the Haskell Standard +Prelude). This means that things like `Int` and `Void` are not part of +the language itself, but are instead part of the standard library. + +Phases of Translation +--------------------- + +> **Commentary** +> +> Because Swift doesn't rely on a C-style "lexer hack" to know what is a +> type and what is a value, it is possible to fully parse a file without +> resolving import declarations. + +Swift has a strict separation between its phases of translation, and the +compiler follows a conceptually simple design. The phases of translation +are: + +- Lexing <langref.lexical>: A source file is broken into tokens + according to a (nearly, `/**/` comments can be nested) + regular grammar. +- Parsing and AST Building: The tokens are parsed according to the + grammar set out below. The grammar is context free and does not + require any "type feedback" from the lexer or later stages. During + parsing, name binding for references to local variables and other + declarations that are not at module (and eventually namespace) scope + are bound. +- Name Binding <langref.namebind>: At this phase, references to + non-local types and values are bound, and import directives + <langref.decl.import> are both validated and searched. Name + binding can cause recursive compilation of modules that are + referenced but not yet built. +- Type Checking <langref.typecheck>: During this phase all types + are resolved within value definitions, function application + <langref.expr.call> and <a href="\#expr-infix">binary + expressions</a> are found and formed, and overloaded functions + are resolved. +- Code Generation: The AST is converted the LLVM IR, optimizations are + performed, and machine code generated. +- Linking: runtime libraries and referenced modules are linked in. + +FIXME: "import Swift" implicitly added as the last import in a source +file. + +Lexical Structure +----------------- + +> **Commentary** +> +> Not all characters are "taken" in the language, this is because it is +> still growing. As there becomes a reason to assign things into the +> identifier or punctuation bucket, we will do so as swift evolves. + +The lexical structure of a Swift file is very simple: the files are +tokenized according to the following productions and categories. As is +usual with most languages, tokenization uses the maximal munch rule and +whitespace separates tokens. This means that "`a b`" and "`ab`" lex into +different token streams and are therefore different in the grammar. + +### Whitespace and Comments + +> **Commentary** +> +> Nested block comments are important because we don't have the nestable +> `#if 0` hack from C to rely on. + +``` {.sourceCode .none} +whitespace ::= ' ' +whitespace ::= '\n' +whitespace ::= '\r' +whitespace ::= '\t' +whitespace ::= '\0' +comment ::= //.*[\n\r] +comment ::= /* .... */ +``` + +Space, newline, tab, and the nul byte are all considered whitespace and +are discarded, with one exception: a '`(`' or '`[`' which does not +follow a non-whitespace character is different kind of token (called +*spaced*) from one which does not (called *unspaced*). A '`(`' or '`[`' +at the beginning of a file is spaced. + +Comments may follow the BCPL style, starting with a "`//`" and running +to the end of the line, or may be recursively nested `/**/` style +comments. Comments are ignored and treated as whitespace. + +### Reserved Punctuation Tokens + +> **Commentary** +> +> Note that `->` is used for function types `() -> Int`, not pointer +> dereferencing. + +``` {.sourceCode .none} +punctuation ::= '(' +punctuation ::= ')' +punctuation ::= '{' +punctuation ::= '}' +punctuation ::= '[' +punctuation ::= ']' +punctuation ::= '.' +punctuation ::= ',' +punctuation ::= ';' +punctuation ::= ':' +punctuation ::= '=' +punctuation ::= '->' +punctuation ::= '&' // unary prefix operator +``` + +These are all reserved punctuation that are lexed into tokens. Most +other non-alphanumeric characters are matched as operators +<langref.lexical.operator>. Unlike operators, these tokens are not +overloadable. + +### Reserved Keywords + +> **Commentary** +> +> The number of keywords is reduced by pushing most functionality into +> the library (e.g. "builtin" datatypes like `Int` and `Bool`). This +> allows us to add new stuff to the library in the future without +> worrying about conflicting with the user's namespace. + +``` {.sourceCode .none} +// Declarations and Type Keywords +keyword ::= 'class' +keyword ::= 'destructor' +keyword ::= 'extension' +keyword ::= 'import' +keyword ::= 'init' +keyword ::= 'func' +keyword ::= 'enum' +keyword ::= 'protocol' +keyword ::= 'struct' +keyword ::= 'subscript' +keyword ::= 'Type' +keyword ::= 'typealias' +keyword ::= 'var' +keyword ::= 'where' + +// Statements +keyword ::= 'break' +keyword ::= 'case' +keyword ::= 'continue' +keyword ::= 'default' +keyword ::= 'do' +keyword ::= 'else' +keyword ::= 'if' +keyword ::= 'in' +keyword ::= 'for' +keyword ::= 'return' +keyword ::= 'switch' +keyword ::= 'then' +keyword ::= 'while' + +// Expressions +keyword ::= 'as' +keyword ::= 'is' +keyword ::= 'new' +keyword ::= 'super' +keyword ::= 'self' +keyword ::= 'Self' +keyword ::= 'type' +keyword ::= '__COLUMN__' +keyword ::= '__FILE__' +keyword ::= '__LINE__' +``` + +These are the builtin keywords. Keywords can still be used as names via +escaped identifiers <langref.lexical.escapedident>. + +### Contextual Keywords + +Swift uses several contextual keywords at various parts of the language. +Contextual keywords are not reserved words, meaning that they can be +used as identifiers. However, in certain contexts, they act as keywords, +and are represented as such in the grammar below. The following +identifiers act as contextual keywords within the language: + +``` {.sourceCode .none} +get +infix +mutating +nonmutating +operator +override +postfix +prefix +set +``` + +### Integer Literals + +``` {.sourceCode .none} +integer_literal ::= [0-9][0-9_]* +integer_literal ::= 0x[0-9a-fA-F][0-9a-fA-F_]* +integer_literal ::= 0o[0-7][0-7_]* +integer_literal ::= 0b[01][01_]* +``` + +Integer literal tokens represent simple integer values of unspecified +precision. They may be expressed in decimal, binary with the '`0b`' +prefix, octal with the '`0o`' prefix, or hexadecimal with the '`0x`' +prefix. Unlike C, a leading zero does not affect the base of the +literal. + +Integer literals may contain underscores at arbitrary positions after +the first digit. These underscores may be used for human readability and +do not affect the value of the literal. + + 789 + 0789 + + 1000000 + 1_000_000 + + 0b111_101_101 + 0o755 + + 0b1111_1011 + 0xFB + +### Floating Point Literals + +> **Commentary** +> +> We require a digit on both sides of the dot to allow lexing "`4.km`" +> as "`4 . km`" instead of "`4. km`" and for a series of dots to be an +> operator (for ranges). The regex for decimal literals is same as Java, +> and the one for hex literals is the same as C99, except that we do not +> allow a trailing suffix that specifies a precision. + +``` {.sourceCode .none} +floating_literal ::= [0-9][0-9_]*\.[0-9][0-9_]* +floating_literal ::= [0-9][0-9_]*\.[0-9][0-9_]*[eE][+-]?[0-9][0-9_]* +floating_literal ::= [0-9][0-9_]*[eE][+-]?[0-9][0-9_]* +floating_literal ::= 0x[0-9A-Fa-f][0-9A-Fa-f_]* + (\.[0-9A-Fa-f][0-9A-Fa-f_]*)?[pP][+-]?[0-9][0-9_]* +``` + +Floating point literal tokens represent floating point values of +unspecified precision. Decimal and hexadecimal floating-point literals +are supported. + +The integer, fraction, and exponent of a floating point literal may each +contain underscores at arbitrary positions after their first digits. +These underscores may be used for human readability and do not affect +the value of the literal. Each part of the floating point literal must +however start with a digit; `1._0` would be a reference to the `_0` +member of `1`. + + 1.0 + 1000000.75 + 1_000_000.75 + + 0x1.FFFFFFFFFFFFFp1022 + 0x1.FFFF_FFFF_FFFF_Fp1_022 + +### Character Literals + +``` {.sourceCode .none} +character_literal ::= '[^'\\\n\r]|character_escape' +character_escape ::= [\]0 [\][\] | [\]t | [\]n | [\]r | [\]" | [\]' +character_escape ::= [\]x hex hex +character_escape ::= [\]u hex hex hex hex +character_escape ::= [\]U hex hex hex hex hex hex hex hex +hex ::= [0-9a-fA-F] +``` + +`character_literal` tokens represent a single character, and are +surrounded by single quotes. + +The ASCII and Unicode character escapes: + +``` {.sourceCode .none} +\0 == nul +\n == new line +\r == carriage return +\t == horizontal tab +\u == small Unicode code points +\U == large Unicode code points +\x == raw ASCII byte (less than 0x80) +``` + +### String Literals + +> **Commentary** +> +> FIXME: Forcing `+` to concatenate strings is somewhat gross, a proper +> protocol would be better. + +``` {.sourceCode .none} +string_literal ::= ["]([^"\\\n\r]|character_escape|escape_expr)*["] +escape_expr ::= [\]escape_expr_body +escape_expr_body ::= [(]escape_expr_body[)] +escape_expr_body ::= [^\n\r"()] +``` + +`string_literal` tokens represent a string, and are surrounded by double +quotes. String literals cannot span multiple lines. + +String literals may contain embedded expressions in them (known as +"interpolated expressions") subject to some specific lexical +constraints: the expression may not contain a double quote \["\], +newline \[n\], or carriage return \[r\]. All parentheses must be +balanced. + +In addition to these lexical rules, an interpolated expression must +satisfy the expr <langref.expr> production of the general swift +grammar. This expression is evaluated, and passed to the constructor for +the inferred type of the string literal. It is concatenated onto any +fixed portions of the string literal with a global "`+`" operator that +is found through normal name lookup. + + // Simple string literal. + "Hello world!" + + // Interpolated expressions. + "\(min)...\(max)" + "Result is \((4+i)*j)" + +### Identifier Tokens + +``` {.sourceCode .none} +identifier ::= id-start id-continue* + +// An identifier can start with an ASCII letter or underscore... +id-start ::= [A-Za-z_] + +// or a Unicode alphanumeric character in the Basic Multilingual Plane... +// (excluding combining characters, which can't appear initially) +id-start ::= [\u00A8\u00AA\u00AD\u00AF\u00B2-\u00B5\u00B7-00BA] +id-start ::= [\u00BC-\u00BE\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u00FF] +id-start ::= [\u0100-\u02FF\u0370-\u167F\u1681-\u180D\u180F-\u1DBF] +id-start ::= [\u1E00-\u1FFF] +id-start ::= [\u200B-\u200D\u202A-\u202E\u203F-\u2040\u2054\u2060-\u206F] +id-start ::= [\u2070-\u20CF\u2100-\u218F\u2460-\u24FF\u2776-\u2793] +id-start ::= [\u2C00-\u2DFF\u2E80-\u2FFF] +id-start ::= [\u3004-\u3007\u3021-\u302F\u3031-\u303F\u3040-\uD7FF] +id-start ::= [\uF900-\uFD3D\uFD40-\uFDCF\uFDF0-\uFE1F\uFE30-FE44] +id-start ::= [\uFE47-\uFFFD] + +// or a non-private-use, valid code point outside of the BMP. +id-start ::= [\u10000-\u1FFFD\u20000-\u2FFFD\u30000-\u3FFFD\u40000-\u4FFFD] +id-start ::= [\u50000-\u5FFFD\u60000-\u6FFFD\u70000-\u7FFFD\u80000-\u8FFFD] +id-start ::= [\u90000-\u9FFFD\uA0000-\uAFFFD\uB0000-\uBFFFD\uC0000-\uCFFFD] +id-start ::= [\uD0000-\uDFFFD\uE0000-\uEFFFD] + +// After the first code point, an identifier can contain ASCII digits... +id-continue ::= [0-9] + +// and/or combining characters... +id-continue ::= [\u0300-\u036F\u1DC0-\u1DFF\u20D0-\u20FF\uFE20-\uFE2F] + +// in addition to the starting character set. +id-continue ::= id-start + +identifier-or-any ::= identifier +identifier-or-any ::= '_' +``` + +The set of valid identifier characters is consistent with WG14 N1518, +"Recommendations for extended identifier characters for C and C++". This +roughly corresponds to the alphanumeric characters in the Basic +Multilingual Plane and all non-private-use code points outside of the +BMP. It excludes mathematical symbols, arrows, line and box drawing +characters, and private-use and invalid code points. An identifier +cannot begin with one of the ASCII digits '0' through '9' or with a +combining character. + +The Swift compiler does not normalize Unicode source code, and matches +identifiers by code points only. Source code must be normalized to a +consistent normalization form before being submitted to the compiler. + + // Valid identifiers + foo + _0 + swift + vernissé + 闪亮 + מבריק + 😄 + + // Invalid identifiers + ☃ // Is a symbol + 0cool // Starts with an ASCII digit + ́foo // Starts with a combining character (U+0301) +  // Is a private-use character (U+F8FF) + +### Operator Tokens + +``` {.sourceCode .none} +operator ::= [/=-+*%<>!&|^~]+ +operator ::= \.+ + +Reserved for punctuation: '.', '=', '->', and unary prefix '&' +Reserved for comments: '//', '/*' and '*/' + +operator-binary ::= operator +operator-prefix ::= operator +operator-postfix ::= operator + +left-binder ::= [ \r\n\t\(\[\{,;:] +right-binder ::= [ \r\n\t\)\]\},;:] + +any-identifier ::= identifier | operator +``` + +`operator-binary`, `operator-prefix`, and `operator-postfix` are +distinguished by immediate lexical context. An operator token is called +*left-bound* if it is immediately preceded by a character matching +`left-binder`. An operator token is called *right-bound* if it is +immediately followed by a character matching `right-binder`. An operator +token is an `operator-prefix` if it is right-bound but not left-bound, +an `operator-postfix` if it is left-bound but not right-bound, and an +`operator-binary` in either of the other two cases. + +As an exception, an operator immediately followed by a dot ('`.`') is +only considered right-bound if not already left-bound. This allows +`a!.prop` to be parsed as `(a!).prop` rather than as `a ! .prop`. + +The '`!`' operator is postfix if it is left-bound. + +The '`?`' operator is postfix (and therefore not the ternary operator) +if it is left-bound. The sugar form for `Optional` types must be +left-bound. + +When parsing certain grammatical constructs that involve '`<`' and '`>`' +(such as <a href="\#type-composition">protocol composition +types</a>), an `operator` with a leading '`<`' or '`>`' may be +split into two or more tokens: the leading '`<`' or '`>`' and the +remainder of the token, which may be an `operator` or `punctuation` +token that may itself be further split. This rule allows us to parse +nested constructs such as `A>` without requiring spaces between the +closing '`>`'s. + +### Implementation Identifier Token + +``` {.sourceCode .none} +dollarident ::= '$' id-continue+ +``` + +Tokens that start with a `$` are separate class of identifier, which are +fixed purpose names that are defined by the implementation. + +### Escaped Identifiers + +``` {.sourceCode .none} +identifier ::= '`' id-start id-continue* '`' +``` + +An identifier that would normally be a +keyword <langref.lexical.keyword> may be used as an identifier by +wrapping it in backticks '`\`', for example:: + + func class\`() { /\* ... \*/ } let type = 0.type + +Any identifier may be escaped, though only identifiers that would +normally be parsed as keywords are required to be. The backtick-quoted +string must still form a valid, non-operator identifier: + + let `0` = 0 // Error, "0" doesn't start with an alphanumeric + let `foo-bar` = 0 // Error, '-' isn't an identifier character + let `+` = 0 // Error, '+' is an operator + +Name Binding +------------ + +Type Checking +------------- + +Declarations +------------ + +... + +### Import Declarations + +... + +### `var` Declarations + +``` {.sourceCode .none} +decl-var-head-kw ::= ('static' | 'class')? 'override'? +decl-var-head-kw ::= 'override'? ('static' | 'class')? + +decl-var-head ::= attribute-list decl-var-head-kw? 'var' + +decl-var ::= decl-var-head pattern initializer? (',' pattern initializer?)* + +// 'get' is implicit in this syntax. +decl-var ::= decl-var-head identifier ':' type brace-item-list + +decl-var ::= decl-var-head identifier ':' type '{' get-set '}' + +decl-var ::= decl-var-head identifier ':' type initializer? '{' willset-didset '}' + +// For use in protocols. +decl-var ::= decl-var-head identifier ':' type '{' get-set-kw '}' + +get-set ::= get set? +get-set ::= set get + +get ::= attribute-list ( 'mutating' | 'nonmutating' )? 'get' brace-item-list +set ::= attribute-list ( 'mutating' | 'nonmutating' )? 'set' set-name? brace-item-list +set-name ::= '(' identifier ')' + +willset-didset ::= willset didset? +willset-didset ::= didset willset? + +willset ::= attribute-list 'willSet' set-name? brace-item-list +didset ::= attribute-list 'didSet' set-name? brace-item-list + +get-kw ::= attribute-list ( 'mutating' | 'nonmutating' )? 'get' +set-kw ::= attribute-list ( 'mutating' | 'nonmutating' )? 'set' +get-set-kw ::= get-kw set-kw? +get-set-kw ::= set-kw get-kw +``` + +`var` declarations form the backbone of value declarations in Swift. A +`var` declaration takes a pattern and an optional initializer, and +declares all the pattern-identifiers in the pattern as variables. If +there is an initializer and the pattern is +fully-typed <langref.types.fully\_typed>, the initializer is +converted to the type of the pattern. If there is an initializer and the +pattern is not fully-typed, the type of initializer is computed +independently of the pattern, and the type of the pattern is derived +from the initializer. If no initializer is specified, the pattern must +be fully-typed, and the values are default-initialized. + +If there is more than one pattern in a `var` declaration, they are each +considered independently, as if there were multiple declarations. The +initial `attribute-list` is shared between all the declared variables. + +A var declaration may contain a getter and (optionally) a setter, which +will be used when reading or writing the variable, respectively. Such a +variable does not have any associated storage. A `var` declaration with +a getter or setter must have a type (call it `T`). The getter function, +whose body is provided as part of the `var-get` clause, has type +`() -> T`. Similarly, the setter function, whose body is part of the +`var-set` clause (if provided), has type `(T) -> ()`. + +If the `var-set` or `willset` clause contains a `set-name` clause, the +identifier of that clause is used as the name of the parameter to the +setter or the observing accessor. Otherwise, the parameter name is +`newValue`. Same applies to `didset` clause, but the default parameter +name is `oldValue`. + +FIXME: Should the type of a pattern which isn't fully typed affect the +type-checking of the expression (i.e. should we compute a structured +dependent type)? + +Like all other declarations, `var`s can optionally have a list of +attributes <langref.decl.attribute\_list> applied to them. + +The type of a variable must be materializable +<langref.types.materializable>. A variable is an lvalue unless it +has a `var-get` clause but not `var-set` clause. + +Here are some examples of `var` declarations: + + // Simple examples. + var a = 4 + var b: Int + var c: Int = 42 + + // This decodes the tuple return value into independently named parts + // and both 'val' and 'err' are in scope after this line. + var (val, err) = foo() + + // Variable getter/setter + var _x: Int = 0 + var x_modify_count: Int = 0 + var x1: Int { + return _x + } + var x2: Int { + get { + return _x + } + set { + x_modify_count = x_modify_count + 1 + _x = value + } + } + +Note that `get`, `set`, `willSet` and `didSet` are context-sensitive +keywords. + +`static` keyword is allowed inside structs and enums, and extensions of +those. + +`class` keyword is allowed inside classes, class extensions, and +protocols. + +> **Ambiguity 1** +> +> The production for implicit `get` makes this grammar ambiguous. For +> example: +> +> class A { +> func get(_: () -> Int) {} +> var a: Int { +> get { return 0 } // Getter declaration or call to 'get' with a trailing closure? +> } +> // But if this was intended as a call to 'get' function, then we have a +> // getter without a 'return' statement, so the code is invalid anyway. +> } +> +> We disambiguate towards `get-set` or `willset-didset` production if +> the first token after `{` is the corresponding keyword, possibly +> preceeded by attributes. Thus, the following code is rejected because +> we are expecting `{` after `set`: +> +> class A { +> var set: Foo +> var a: Int { +> set.doFoo() +> return 0 +> } +> } + +> **Ambiguity 2** +> +> The production with `initializer` and an accessor block is ambiguous. +> For example: +> +> func takeClosure(_: () -> Int) {} +> struct A { +> var willSet: Int +> var a: Int = takeClosure { +> willSet {} // A 'willSet' declaration or a call to 'takeClosure'? +> } +> } +> +> We disambiguate towards `willget-didset` production if the first token +> after `{` is the keyword `willSet` or `didSet`, possibly preceeded by +> attributes. + +> **Rationale** +> +> Even though it is possible to do further checks and speculatively +> parse more, it introduces unjustified complexity to cover (hopefully +> rare) corner cases. In ambiguous cases users can always opt-out of the +> trailing closure syntax by using explicit parentheses in the function +> call. + +### `func` Declarations + +``` {.sourceCode .none} +// Keywords can be specified in any order. +decl-func-head-kw ::= ( 'static' | 'class' )? 'override'? ( 'mutating' | 'nonmutating' )? + +decl-func ::= attribute-list decl-func-head-kw? 'func' any-identifier generic-params? func-signature brace-item-list? +``` + +`func` is a declaration for a function. The argument list and optional +return value are specified by the type production of the function, and +the body is either a brace expression or elided. Like all other +declarations, functions are can have attributes. + +If the type is not syntactically a function type (i.e., has no `->` in +it at top-level), then the return value is implicitly inferred to be +`()`. All of the argument and return value names are injected into the +<a href="\#namebind\_scope">scope</a> of the function +body.</p> + +A function in an <a href="\#decl-extension">extension</a> of +some type (or in other places that are semantically equivalent to an +extension) implicitly get a `self` argument with these rules ... +\[todo\] + +`static` and `class` functions are only allowed in an <a +href="\#decl-extension">extension</a> of some type (or in other +places that are semantically equivalent to an extension). They indicate +that the function is actually defined on the <a +href="\#metatype">metatype</a> for the type, not on the type +itself. Thus its implicit `self` argument is actually of metatype type. + +`static` keyword is allowed inside structs and enums, and extensions of +those. + +`class` keyword is allowed inside classes, class extensions, and +protocols. + +TODO: Func should be an immutable name binding, it should implicitly add +an attribute immutable when it exists. + +TODO: Incoming arguments should be readonly, result should be implicitly +writeonly when we have these attributes. + +#### Function signatures + +... + +An argument name is a keyword argument if: - It is an argument to an +initializer, or - It is an argument to a method after the first +argument, or - It is preceded by a back-tick (), or +- Both a keyword argument name and an internal parameter name are specified. + +.. \_langref.decl.subscript: + +subscript\` Declarations --------------------------- + +``` {.sourceCode .none} +subscript-head ::= attribute-list 'override'? 'subscript' pattern-tuple '->' type + +decl-subscript ::= subscript-head '{' get-set '}' + +// 'get' is implicit in this syntax. +decl-subscript ::= subscript-head brace-item-list + +// For use in protocols. +decl-subscript ::= subscript-head '{' get-set-kw '}' +``` + +A subscript declaration provides support for <a +href="\#expr-subscript"> subscripting</a> an object of a +particular type via a getter and (optional) setter. Therefore, subscript +declarations can only appear within a type definition or extension. + +The `pattern-tuple` of a subscript declaration provides the indices that +will be used in the subscript expression, e.g., the `i` in `a[i]`. This +pattern must be fully-typed. The `type` following the arrow provides the +type of element being accessed, which must be materializable. Subscript +declarations can be overloaded, so long as either the `pattern-tuple` or +`type` differs from other declarations. + +The `get-set` clause specifies the getter and setter used for +subscripting. The getter is a function whose input is the type of the +`pattern-tuple` and whose result is the element type. Similarly, the +setter is a function whose result type is `()` and whose input is the +type of the `pattern-tuple` with a parameter of the element type added +to the end of the tuple; the name of the parameter is the `set-name`, if +provided, or `value` otherwise. + + // Simple bit vector with storage for 64 boolean values + struct BitVector64 { + var bits: Int64 + + // Allow subscripting with integer subscripts and a boolean result. + subscript (bit : Int) -> Bool { + // Getter tests the given bit + get { + return bits & (1 << bit)) != 0 + } + + // Setter sets the given bit to the provided value. + set { + var mask = 1 << bit + if value { + bits = bits | mask + } else { + bits = bits & ~mask + } + } + } + } + + var vec = BitVector64() + vec[2] = true + if vec[3] { + print("third bit is set") + } + +### Attribute Lists + +... + +Types +----- + +... + +### Fully-Typed Types + +... + +### Materializable Types + +... + +Patterns +-------- + +> **Commentary** +> +> The pattern grammar mirrors the expression grammar, or to be more +> specific, the grammar of literals. This is because the conceptual +> algorithm for matching a value against a pattern is to try to find an +> assignment of values to variables which makes the pattern equal the +> value. So every expression form which can be used to build a value +> directly should generally have a corresponding pattern form. + +``` {.sourceCode .none} +pattern-atom ::= pattern-var +pattern-atom ::= pattern-any +pattern-atom ::= pattern-tuple +pattern-atom ::= pattern-is +pattern-atom ::= pattern-enum-element +pattern-atom ::= expr + +pattern ::= pattern-atom +pattern ::= pattern-typed +``` + +A pattern represents the structure of a composite value. Parts of a +value can be extracted and bound to variables or compared against other +values by *pattern matching*. Among other places, pattern matching +occurs on the left-hand side of var bindings <langref.decl.var>, +in the arguments of func declarations <langref.decl.func>, and in +the <tt>case</tt> labels of +switch statements <langref.stmt.switch>. Some examples: + + var point = (1, 0, 0) + + // Extract the elements of the "point" tuple and bind them to + // variables x, y, and z. + var (x, y, z) = point + print("x=\(x) y=\(y) z=\(z)") + + // Dispatch on the elements of a tuple in a "switch" statement. + switch point { + case (0, 0, 0): + print("origin") + // The pattern "_" matches any value. + case (_, 0, 0): + print("on the x axis") + case (0, _, 0): + print("on the y axis") + case (0, 0, _): + print("on the z axis") + case (var x, var y, var z): + print("x=\(x) y=\(y) z=\(z)") + } + +A pattern may be "irrefutable", meaning informally that it matches all +values of its type. Patterns in declarations, such as +var <langref.decl.var> and func <langref.decl.func>, are +required to be irrefutable. Patterns in the `case` labels of +switch statements <langref.stmt.switch>, however, are not. + +The basic pattern grammar is a literal "atom" followed by an optional +type annotation. Type annotations are useful for documentation, as well +as for coercing a matched expression to a particular kind. They are also +required when patterns are used in a function signature +<langref.decl.func.signature>. Type annotations are currently not +allowed in switch statements. + +A pattern has a type. A pattern may be "fully-typed", meaning informally +that its type is fully determined by the type annotations it contains. +Some patterns may also derive a type from their context, be it an +enclosing pattern or the way it is used; this set of situations is not +yet fully determined. + +### Typed Patterns + +``` {.sourceCode .none} +pattern-typed ::= pattern-atom ':' type +``` + +A type annotation constrains a pattern to have a specific type. An +annotated pattern is fully-typed if its annotation type is fully-typed. +It is irrefutable if and only if its subpattern is irrefutable. + +Type annotations are currently not allowed in the `case` labels of +`switch` statements; case patterns always get their type from the +subject of the switch. + +### Any Patterns + +``` {.sourceCode .none} +pattern-any ::= '_' +``` + +The symbol `_` in a pattern matches and ignores any value. It is +irrefutable. + +### 'var' and 'let' Patterns + +``` {.sourceCode .none} +pattern-var ::= 'let' pattern +pattern-var ::= 'var' pattern +``` + +The `var` and `let` keywords within a pattern introduces variable +bindings. Any identifiers within the subpattern bind new named variables +to their matching values. 'var' bindings are mutable within the bound +scope, and 'let' bindings are immutable. + + var point = (0, 0, 0) + switch point { + // Bind x, y, z to the elements of point. + case (var x, var y, var z): + print("x=\(x) y=\(y) z=\(z)") + } + + switch point { + // Same. 'var' distributes to the identifiers in its subpattern. + case var (x, y, z): + print("x=\(x) y=\(y) z=\(z)") + } + +Outside of a <tt>var</tt> pattern, an identifier behaves as +an expression +pattern <langref.pattern.expr> referencing an existing definition. + + var zero = 0 + switch point { + // x and z are bound as new variables. + // zero is a reference to the existing 'zero' variable. + case (var x, zero, var z): + print("point off the y axis: x=\(x) z=\(z)") + default: + print("on the y axis") + } + +The left-hand pattern of a var declaration <langref.decl.var> and +the argument pattern of a func declaration <langref.decl.func> are +implicitly inside a `var` pattern; identifiers in their patterns always +bind variables. Variable bindings are irrefutable. + +The type of a bound variable must be materializable +<langref.types.materializable> unless it appears in a +func-signature +<langref.decl.func.signature> and is directly of a +`inout`-annotated type. + +### Tuple Patterns + +``` {.sourceCode .none} +pattern-tuple ::= '(' pattern-tuple-body? ')' +pattern-tuple-body ::= pattern-tuple-element (',' pattern-tuple-body)* '...'? +pattern-tuple-element ::= pattern +``` + +A tuple pattern is a list of zero or more patterns. Within a function +signature <langref.decl.func.signature>, patterns may also be +given a default-value expression. + +A tuple pattern is irrefutable if all its sub-patterns are irrefutable. + +A tuple pattern is fully-typed if all its sub-patterns are fully-typed, +in which case its type is the corresponding tuple type, where each +`type-tuple-element` has the type, label, and default value of the +corresponding `pattern-tuple-element`. A `pattern-tuple-element` has a +label if it is a named pattern or a type annotation of a named pattern. + +A tuple pattern whose body ends in `'...'` is a varargs tuple. The last +element of such a tuple must be a typed pattern, and the type of that +pattern is changed from `T` to `T[]`. The corresponding tuple type for a +varargs tuple is a varargs tuple type. + +As a special case, a tuple pattern with one element that has no label, +has no default value, and is not varargs is treated as a grouping +parenthesis: it has the type of its constituent pattern, not a tuple +type. + +### `is` Patterns + +``` {.sourceCode .none} +pattern-is ::= 'is' type +``` + +`is` patterns perform a type check equivalent to the `x is T` <a +href="\#expr-cast">cast operator</a>. The pattern matches if +the runtime type of a value is of the given type. `is` patterns are +refutable and thus cannot appear in declarations. + + class B {} + class D1 : B {} + class D2 : B {} + + var bs : B[] = [B(), D1(), D2()] + + for b in bs { + switch b { + case is B: + print("B") + case is D1: + print("D1") + case is D2: + print("D2") + } + } + +### Enum Element Patterns + +``` {.sourceCode .none} +pattern-enum-element ::= type-identifier? '.' identifier pattern-tuple? +``` + +Enum element patterns match a value of <a href="\#type-enum">enum +type</a> if the value matches the referenced `case` of the enum. +If the `case` has a type, the value of that type can be matched against +an optional subpattern. + + enum HTMLTag { + case A(href: String) + case IMG(src: String, alt: String) + case BR + } + + switch tag { + case .BR: + print("
") + case .IMG(var src, var alt): + print("\"\(escape(alt))\"") + case .A(var href): + print("") + } + +Enum element patterns are refutable and thus cannot appear in +declarations. (They are currently considered refutable even if the enum +contains only a single `case`.) + +### Expressions in Patterns + +Patterns may include arbitrary expressions as subpatterns. Expression +patterns are refutable and thus cannot appear in declarations. An +expression pattern is compared to its corresponding value using the `~=` +operator. The match succeeds if `expr ~= value` evaluates to true. The +standard library provides a default implementation of `~=` using `==` +equality; additionally, range objects may be matched against integer and +floating-point values. The `~=` operator may be overloaded like any +function. + + var point = (0, 0, 0) + switch point { + // Equality comparison. + case (0, 0, 0): + print("origin") + // Range comparison. + case (-10...10, -10...10, -10...10): + print("close to the origin") + default: + print("too far away") + } + + // Define pattern matching of an integer value to a string expression. + func ~=(pattern:String, value:Int) -> Bool { + return pattern == "\(value)" + } + + // Now we can pattern-match strings to integers: + switch point { + case ("0", "0", "0"): + print("origin") + default: + print("not the origin") + } + +The order of evaluation of expressions in patterns, including whether an +expression is evaluated at all, is unspecified. The compiler is free to +reorder or elide expression evaluation in patterns to improve dispatch +efficiency. Expressions in patterns therefore cannot be relied on for +side effects. + +Expressions +----------- + +... + +### Function Application + +... + +Statements +---------- + +... + +### `break` Statement + +``` {.sourceCode .none} +stmt-return ::= 'break' +``` + +The 'break' statement transfers control out of the enclosing 'for' loop +or 'while' loop. + +### `continue` Statement + +``` {.sourceCode .none} +stmt-return ::= 'continue' +``` + +The 'continue' statement transfers control back to the start of the +enclosing 'for' loop or 'while' loop. + +... + +### `switch` Statement + +``` {.sourceCode .none} +stmt-switch ::= 'switch' expr-basic '{' stmt-switch-case* '}' + +stmt-switch-case ::= (case-label | default-label) brace-item+ +stmt-switch-case ::= (case-label | default-label) ';' + +case-label ::= 'case' pattern ('where' expr)? (',' pattern ('where' expr)?)* ':' +default-label ::= 'default' ':' +``` + +'switch' statements branch on the value of an expression by pattern +matching <langref.pattern>. The subject expression of the switch +is evaluated and tested against the patterns in its `case` labels in +source order. When a pattern is found that matches the value, control is +transferred into the matching `case` block. `case` labels may declare +multiple patterns separated by commas. Only a single `case` labels may +precede a block of code. Case labels may optionally specify a *guard* +expression, introduced by the `where` keyword; if present, control is +transferred to the case only if the subject value both matches the +corresponding pattern and the guard expression evaluates to true. +Patterns are tested "as if" in source order; if multiple cases can match +a value, control is transferred only to the first matching case. The +actual execution order of pattern matching operations, and in particular +the evaluation order of expression patterns +<langref.pattern.expr>, is unspecified. + +A switch may also contain a `default` block. If present, it receives +control if no cases match the subject value. The `default` block must +appear at the end of the switch and must be the only label for its +block. `default` is equivalent to a final `case _` pattern. Switches are +required to be exhaustive; either the contained case patterns must cover +every possible value of the subject's type, or else an explicit +`default` block must be specified to handle uncovered cases. + +Every case and default block has its own scope. Declarations within a +case or default block are only visible within that block. Case patterns +may bind variables using the var keyword <langref.pattern.var>; +those variables are also scoped into the corresponding case block, and +may be referenced in the `where` guard for the case label. However, if a +case block matches multiple patterns, none of those patterns may contain +variable bindings. + +Control does not implicitly 'fall through' from one case block to the +next. fallthrough statements <langref.stmt.fallthrough> may +explicitly transfer control among case blocks. +break <langref.stmt.break> and +continue <langref.stmt.continue> within a switch will break or +continue out of an enclosing 'while' or 'for' loop, not out of the +'switch' itself. + +At least one `brace-item` is required in every case or default block. It +is allowed to be a no-op. Semicolon can be used as a single no-op +statement in otherwise empty cases in switch statements. + + func classifyPoint(point: (Int, Int)) { + switch point { + case (0, 0): + print("origin") + + case (_, 0): + print("on the x axis") + + case (0, _): + print("on the y axis") + + case (var x, var y) where x == y: + print("on the y = x diagonal") + + case (var x, var y) where -x == y: + print("on the y = -x diagonal") + + case (var x, var y): + print("length \(sqrt(x*x + y*y))") + } + } + + switch x { + case 1, 2, 3: + print("x is 1, 2 or 3") + default: + ; + } + +### `fallthrough` Statement + +``` {.sourceCode .none} +stmt-fallthrough ::= 'fallthrough' +``` + +`fallthrough` transfers control from a `case` block of a switch +statement <langref.stmt.switch> to the next `case` or `default` +block within the switch. It may only appear inside a `switch`. +`fallthrough` cannot be used in the final block of a `switch`. It also +cannot transfer control into a `case` block whose pattern contains +var bindings +<langref.pattern.var>. + +Standard Library +---------------- + +> **Commentary** +> +> It would be really great to have literate swift code someday, that way +> this could be generated directly from the code. This would also be +> powerful for Swift library developers to be able to depend on being +> available and standardized. + +This describes some of the standard swift code as it is being built up. +Since Swift is designed to give power to the library developers, much of +what is normally considered the "language" is actually just implemented +in the library. + +All of this code is published by the '`swift`' module, which is +implicitly imported into each source file, unless some sort of pragma in +the code (attribute on an import?) is used to change or disable this +behavior. + +Builtin Module +-------------- + +In the initial Swift implementation, a module named `Builtin` is +imported into every file. Its declarations can only be found by <a +href="\#expr-dot">dot syntax</a>. It provides access to a small +number of primitive representation types and operations defined over +them that map directly to LLVM IR. + +The existence of and details of this module are a private implementation +detail used by our implementation of the standard library. Swift code +outside the standard library should not be aware of this library, and an +independent implementation of the swift standard library should be +allowed to be implemented without the builtin library if it desires. + +For reference below, the description of the standard library uses the +"`Builtin.`" namespace to refer to this module, but independent +implementations could use another implementation if they so desire. + +### Simple Types + +#### Void + + // Void is just a type alias for the empty tuple. + typealias Void = () + +#### Int, Int8, Int16, Int32, Int64 + +> **Commentary** +> +> Having a single standardized integer type that can be used by default +> everywhere is important. One advantage Swift has is that by the time +> it is in widespread use, 64-bit architectures will be pervasive, and +> the LLVM optimizer should grow to be good at shrinking 64-bit integers +> to 32-bit in many cases for those 32-bit architectures that persist. + + // Fixed size types are simple structs of the right size. + struct Int8 { value : Builtin.Int8 } + struct Int16 { value : Builtin.Int16 } + struct Int32 { value : Builtin.Int32 } + struct Int64 { value : Builtin.Int64 } + struct Int128 { value : Builtin.Int128 } + + // Int is just an alias for the 64-bit integer type. + typealias Int = Int64 + +#### Int, Int8, Int16, Int32, Int64 + + struct Float { value : Builtin.FPIEEE32 } + struct Double { value : Builtin.FPIEEE64 } + +#### Bool, true, false + + // Bool is a simple enum. + enum Bool { + true, false + } + + // Allow true and false to be used unqualified. + var true = Bool.true + var false = Bool.false + +### Arithmetic and Logical Operations + +#### Arithmetic Operators + + func * (lhs: Int, rhs: Int) -> Int + func / (lhs: Int, rhs: Int) -> Int + func % (lhs: Int, rhs: Int) -> Int + func + (lhs: Int, rhs: Int) -> Int + func - (lhs: Int, rhs: Int) -> Int + +#### Relational and Equality Operators + + func < (lhs : Int, rhs : Int) -> Bool + func > (lhs : Int, rhs : Int) -> Bool + func <= (lhs : Int, rhs : Int) -> Bool + func >= (lhs : Int, rhs : Int) -> Bool + func == (lhs : Int, rhs : Int) -> Bool + func != (lhs : Int, rhs : Int) -> Bool + +#### Short Circuiting Logical Operators + + func && (lhs: Bool, rhs: ()->Bool) -> Bool + func || (lhs: Bool, rhs: ()->Bool) -> Bool + +Swift has a simplified precedence levels when compared with C. From +highest to lowest: + + "exponentiative:" <<, >> + "multiplicative:" *, /, %, & + "additive:" +, -, |, ^ + "comparative:" ==, !=, <, <=, >=, > + "conjunctive:" && + "disjunctive:" || diff --git a/docs/archive/LangRefNew.rst b/docs/archive/LangRefNew.rst deleted file mode 100644 index bd486e9fd88c9..0000000000000 --- a/docs/archive/LangRefNew.rst +++ /dev/null @@ -1,1555 +0,0 @@ -:orphan: - -.. @raise litre.TestsAreMissing - -=============================== -Swift Language Reference Manual -=============================== - -.. contents:: - -Introduction -============ - -.. warning:: This document has not been kept up to date. - -.. admonition:: Commentary - - In addition to the main spec, there are lots of open ended questions, - justification, and ideas of what best practices should be. That random - discussion is placed in boxes like this one to clarify what is normative and - what is discussion. - -This is the language reference manual for the Swift language, which is highly -volatile and constantly under development. As the prototype evolves, this -document should be kept up to date with what is actually implemented. - -The grammar and structure of the language is defined in BNF form in yellow -boxes. Examples are shown in gray boxes, and assume that the standard library -is in use (unless otherwise specified). - -Basic Goals ------------ - -.. admonition:: Commentary - - A non-goal of the Swift project in general is to become some amazing research - project. We really want to focus on delivering a real product, and having - the design and spec co-evolve. - -In no particular order, and not explained well: - -* Support building great frameworks and applications, with a specific focus on - permiting rich and powerful APIs. -* Get the defaults right: this reduces the barrier to entry and increases the - odds that the right thing happens. -* Through our support for building great APIs, we aim to provide an expressive - and productive language that is fun to program in. -* Support low-level system programming. We should want to write compilers, - operating system kernels, and media codecs in Swift. This means that being - able to obtain high performance is really quite important. -* Provide really great tools, like an IDE, debugger, profiling, etc. -* Where possible, steal great ideas instead of innovating new things that will - work out in unpredictable ways. It turns out that there are a lot of good - ideas already out there. -* Memory safe by default: array overrun errors, uninitialized values, and other - problems endemic to C should not occur in Swift, even if it means some amount - of runtime overhead. Eventually these checks will be disablable for people - who want ultimate performance in production builds. -* Efficiently implementable with a static compiler: runtime compilation is - great technology and Swift may eventually get a runtime optimizer, but it is - a strong goal to be able to implement swift with just a static compiler. -* Interoperate as transparently as possible with C, Objective-C, and C++ - without having to write an equivalent of "extern C" for every referenced - definition. -* Great support for efficient by-value types. -* Elegant and natural syntax, aiming to be familiar and easy to transition to - for "C" people. Differences from the C family should only be done when it - provides a significant win (e.g. eliminate declarator syntax). -* Lots of other stuff too. - -A smaller wishlist goal is to support embedded sub-languages in swift, so that -we don't get the OpenCL-is-like-C-but-very-different-in-many-details -problem. - -Basic Approach --------------- - -.. admonition:: Commentary - - Pushing as much of the language as realistic out of the compiler and into the - library is generally good for a few reasons: 1) we end up with a smaller core - language. 2) we force the language that is left to be highly expressive and - extensible. 3) this highly expressive language core can then be used to - build a lot of other great libraries, hopefully many we can't even anticipate - at this point. - -The basic approach in designing and implementing the Swift prototype was to -start at the very bottom of the stack (simple expressions and the trivial bits -of the type system) and incrementally build things up one brick at a time. -There is a big focus on making things as simple as possible and having a clean -internal core. Where it makes sense, sugar is added on top to make the core -more expressive for common situations. - -One major aspect that dovetails with expressivity, learnability, and focus on -API development is that much of the language is implemented in a :ref:`standard -library ` (inspired in part by the Haskell Standard Prelude). -This means that things like ``Int`` and ``Void`` are not part of the language -itself, but are instead part of the standard library. - -Phases of Translation -===================== - -.. admonition:: Commentary - - Because Swift doesn't rely on a C-style "lexer hack" to know what is a type - and what is a value, it is possible to fully parse a file without resolving - import declarations. - -Swift has a strict separation between its phases of translation, and the -compiler follows a conceptually simple design. The phases of translation -are: - -* :ref:`Lexing `: A source file is broken into tokens - according to a (nearly, ``/**/`` comments can be nested) regular grammar. - -* Parsing and AST Building: The tokens are parsed according to the grammar set - out below. The grammar is context free and does not require any "type - feedback" from the lexer or later stages. During parsing, name binding for - references to local variables and other declarations that are not at module - (and eventually namespace) scope are bound. - -* :ref:`Name Binding `: At this phase, references to - non-local types and values are bound, and :ref:`import directives - ` are both validated and searched. Name binding can - cause recursive compilation of modules that are referenced but not yet built. - -* :ref:`Type Checking `: During this phase all types are - resolved within value definitions, :ref:`function application - ` and binary expressions are - found and formed, and overloaded functions are resolved. - -* Code Generation: The AST is converted the LLVM IR, optimizations are - performed, and machine code generated. - -* Linking: runtime libraries and referenced modules are linked in. - -FIXME: "import Swift" implicitly added as the last import in a source file. - -.. _langref.lexical: - -Lexical Structure -================= - -.. admonition:: Commentary - - Not all characters are "taken" in the language, this is because it is still - growing. As there becomes a reason to assign things into the identifier or - punctuation bucket, we will do so as swift evolves. - -The lexical structure of a Swift file is very simple: the files are tokenized -according to the following productions and categories. As is usual with most -languages, tokenization uses the maximal munch rule and whitespace separates -tokens. This means that "``a b``" and "``ab``" lex into different token -streams and are therefore different in the grammar. - -.. _langref.lexical.whitespace: - -Whitespace and Comments ------------------------ - -.. admonition:: Commentary - - Nested block comments are important because we don't have the nestable ``#if - 0`` hack from C to rely on. - -.. code-block:: none - - whitespace ::= ' ' - whitespace ::= '\n' - whitespace ::= '\r' - whitespace ::= '\t' - whitespace ::= '\0' - comment ::= //.*[\n\r] - comment ::= /* .... */ - -Space, newline, tab, and the nul byte are all considered whitespace and are -discarded, with one exception: a '``(``' or '``[``' which does not follow a -non-whitespace character is different kind of token (called *spaced*) -from one which does not (called *unspaced*). A '``(``' or '``[``' at the -beginning of a file is spaced. - -Comments may follow the BCPL style, starting with a "``//``" and running to the -end of the line, or may be recursively nested ``/**/`` style comments. Comments -are ignored and treated as whitespace. - -.. _langref.lexical.reserved_punctuation: - -Reserved Punctuation Tokens ---------------------------- - -.. admonition:: Commentary - - Note that ``->`` is used for function types ``() -> Int``, not pointer - dereferencing. - -.. code-block:: none - - punctuation ::= '(' - punctuation ::= ')' - punctuation ::= '{' - punctuation ::= '}' - punctuation ::= '[' - punctuation ::= ']' - punctuation ::= '.' - punctuation ::= ',' - punctuation ::= ';' - punctuation ::= ':' - punctuation ::= '=' - punctuation ::= '->' - punctuation ::= '&' // unary prefix operator - -These are all reserved punctuation that are lexed into tokens. Most other -non-alphanumeric characters are matched as :ref:`operators -`. Unlike operators, these tokens are not -overloadable. - -.. _langref.lexical.keyword: - -Reserved Keywords ------------------ - -.. admonition:: Commentary - - The number of keywords is reduced by pushing most functionality into the - library (e.g. "builtin" datatypes like ``Int`` and ``Bool``). This allows us - to add new stuff to the library in the future without worrying about - conflicting with the user's namespace. - -.. code-block:: none - - // Declarations and Type Keywords - keyword ::= 'class' - keyword ::= 'destructor' - keyword ::= 'extension' - keyword ::= 'import' - keyword ::= 'init' - keyword ::= 'func' - keyword ::= 'enum' - keyword ::= 'protocol' - keyword ::= 'struct' - keyword ::= 'subscript' - keyword ::= 'Type' - keyword ::= 'typealias' - keyword ::= 'var' - keyword ::= 'where' - - // Statements - keyword ::= 'break' - keyword ::= 'case' - keyword ::= 'continue' - keyword ::= 'default' - keyword ::= 'do' - keyword ::= 'else' - keyword ::= 'if' - keyword ::= 'in' - keyword ::= 'for' - keyword ::= 'return' - keyword ::= 'switch' - keyword ::= 'then' - keyword ::= 'while' - - // Expressions - keyword ::= 'as' - keyword ::= 'is' - keyword ::= 'new' - keyword ::= 'super' - keyword ::= 'self' - keyword ::= 'Self' - keyword ::= 'type' - keyword ::= '__COLUMN__' - keyword ::= '__FILE__' - keyword ::= '__LINE__' - - -These are the builtin keywords. Keywords can still be used as names via -`escaped identifiers `. - -Contextual Keywords -------------------- - -Swift uses several contextual keywords at various parts of the language. -Contextual keywords are not reserved words, meaning that they can be used as -identifiers. However, in certain contexts, they act as keywords, and are -represented as such in the grammar below. The following identifiers act as -contextual keywords within the language: - -.. code-block:: none - - get - infix - mutating - nonmutating - operator - override - postfix - prefix - set - -.. _langref.lexical.integer_literal: - -Integer Literals ----------------- - -.. code-block:: none - - integer_literal ::= [0-9][0-9_]* - integer_literal ::= 0x[0-9a-fA-F][0-9a-fA-F_]* - integer_literal ::= 0o[0-7][0-7_]* - integer_literal ::= 0b[01][01_]* - -Integer literal tokens represent simple integer values of unspecified -precision. They may be expressed in decimal, binary with the '``0b``' prefix, -octal with the '``0o``' prefix, or hexadecimal with the '``0x``' prefix. -Unlike C, a leading zero does not affect the base of the literal. - -Integer literals may contain underscores at arbitrary positions after the first -digit. These underscores may be used for human readability and do not affect -the value of the literal. - -:: - - 789 - 0789 - - 1000000 - 1_000_000 - - 0b111_101_101 - 0o755 - - 0b1111_1011 - 0xFB - -.. _langref.lexical.floating_literal: - -Floating Point Literals ------------------------ - -.. admonition:: Commentary - - We require a digit on both sides of the dot to allow lexing "``4.km``" as - "``4 . km``" instead of "``4. km``" and for a series of dots to be an - operator (for ranges). The regex for decimal literals is same as Java, and - the one for hex literals is the same as C99, except that we do not allow a - trailing suffix that specifies a precision. - -.. code-block:: none - - floating_literal ::= [0-9][0-9_]*\.[0-9][0-9_]* - floating_literal ::= [0-9][0-9_]*\.[0-9][0-9_]*[eE][+-]?[0-9][0-9_]* - floating_literal ::= [0-9][0-9_]*[eE][+-]?[0-9][0-9_]* - floating_literal ::= 0x[0-9A-Fa-f][0-9A-Fa-f_]* - (\.[0-9A-Fa-f][0-9A-Fa-f_]*)?[pP][+-]?[0-9][0-9_]* - -Floating point literal tokens represent floating point values of unspecified -precision. Decimal and hexadecimal floating-point literals are supported. - -The integer, fraction, and exponent of a floating point literal may each -contain underscores at arbitrary positions after their first digits. These -underscores may be used for human readability and do not affect the value of -the literal. Each part of the floating point literal must however start with a -digit; ``1._0`` would be a reference to the ``_0`` member of ``1``. - -:: - - 1.0 - 1000000.75 - 1_000_000.75 - - 0x1.FFFFFFFFFFFFFp1022 - 0x1.FFFF_FFFF_FFFF_Fp1_022 - -.. _langref.lexical.character_literal: - -Character Literals ------------------- - -.. code-block:: none - - character_literal ::= '[^'\\\n\r]|character_escape' - character_escape ::= [\]0 [\][\] | [\]t | [\]n | [\]r | [\]" | [\]' - character_escape ::= [\]x hex hex - character_escape ::= [\]u hex hex hex hex - character_escape ::= [\]U hex hex hex hex hex hex hex hex - hex ::= [0-9a-fA-F] - -``character_literal`` tokens represent a single character, and are surrounded -by single quotes. - -The ASCII and Unicode character escapes: - -.. code-block:: none - - \0 == nul - \n == new line - \r == carriage return - \t == horizontal tab - \u == small Unicode code points - \U == large Unicode code points - \x == raw ASCII byte (less than 0x80) - -.. _langref.lexical.string_literal: - -String Literals ---------------- - -.. admonition:: Commentary - - FIXME: Forcing ``+`` to concatenate strings is somewhat gross, a proper protocol - would be better. - -.. code-block:: none - - string_literal ::= ["]([^"\\\n\r]|character_escape|escape_expr)*["] - escape_expr ::= [\]escape_expr_body - escape_expr_body ::= [(]escape_expr_body[)] - escape_expr_body ::= [^\n\r"()] - -``string_literal`` tokens represent a string, and are surrounded by double -quotes. String literals cannot span multiple lines. - -String literals may contain embedded expressions in them (known as -"interpolated expressions") subject to some specific lexical constraints: the -expression may not contain a double quote ["], newline [\n], or carriage return -[\r]. All parentheses must be balanced. - -In addition to these lexical rules, an interpolated expression must satisfy the -:ref:`expr ` production of the general swift grammar. This -expression is evaluated, and passed to the constructor for the inferred type of -the string literal. It is concatenated onto any fixed portions of the string -literal with a global "``+``" operator that is found through normal name -lookup. - -:: - - // Simple string literal. - "Hello world!" - - // Interpolated expressions. - "\(min)...\(max)" + "Result is \((4+i)*j)" - -.. _langref.lexical.identifier: - -Identifier Tokens ------------------ - -.. code-block:: none - - identifier ::= id-start id-continue* - - // An identifier can start with an ASCII letter or underscore... - id-start ::= [A-Za-z_] - - // or a Unicode alphanumeric character in the Basic Multilingual Plane... - // (excluding combining characters, which can't appear initially) - id-start ::= [\u00A8\u00AA\u00AD\u00AF\u00B2-\u00B5\u00B7-00BA] - id-start ::= [\u00BC-\u00BE\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u00FF] - id-start ::= [\u0100-\u02FF\u0370-\u167F\u1681-\u180D\u180F-\u1DBF] - id-start ::= [\u1E00-\u1FFF] - id-start ::= [\u200B-\u200D\u202A-\u202E\u203F-\u2040\u2054\u2060-\u206F] - id-start ::= [\u2070-\u20CF\u2100-\u218F\u2460-\u24FF\u2776-\u2793] - id-start ::= [\u2C00-\u2DFF\u2E80-\u2FFF] - id-start ::= [\u3004-\u3007\u3021-\u302F\u3031-\u303F\u3040-\uD7FF] - id-start ::= [\uF900-\uFD3D\uFD40-\uFDCF\uFDF0-\uFE1F\uFE30-FE44] - id-start ::= [\uFE47-\uFFFD] - - // or a non-private-use, valid code point outside of the BMP. - id-start ::= [\u10000-\u1FFFD\u20000-\u2FFFD\u30000-\u3FFFD\u40000-\u4FFFD] - id-start ::= [\u50000-\u5FFFD\u60000-\u6FFFD\u70000-\u7FFFD\u80000-\u8FFFD] - id-start ::= [\u90000-\u9FFFD\uA0000-\uAFFFD\uB0000-\uBFFFD\uC0000-\uCFFFD] - id-start ::= [\uD0000-\uDFFFD\uE0000-\uEFFFD] - - // After the first code point, an identifier can contain ASCII digits... - id-continue ::= [0-9] - - // and/or combining characters... - id-continue ::= [\u0300-\u036F\u1DC0-\u1DFF\u20D0-\u20FF\uFE20-\uFE2F] - - // in addition to the starting character set. - id-continue ::= id-start - - identifier-or-any ::= identifier - identifier-or-any ::= '_' - -The set of valid identifier characters is consistent with WG14 N1518, -"Recommendations for extended identifier characters for C and C++". This -roughly corresponds to the alphanumeric characters in the Basic Multilingual -Plane and all non-private-use code points outside of the BMP. It excludes -mathematical symbols, arrows, line and box drawing characters, and private-use -and invalid code points. An identifier cannot begin with one of the ASCII -digits '0' through '9' or with a combining character. - -The Swift compiler does not normalize Unicode source code, and matches -identifiers by code points only. Source code must be normalized to a consistent -normalization form before being submitted to the compiler. - -:: - - // Valid identifiers - foo - _0 - swift - vernissé - 闪亮 - מבריק - 😄 - - // Invalid identifiers - ☃ // Is a symbol - 0cool // Starts with an ASCII digit - ́foo // Starts with a combining character (U+0301) -  // Is a private-use character (U+F8FF) - -.. _langref.lexical.operator: - -Operator Tokens ---------------- - -.. code-block:: none - - operator ::= [/=-+*%<>!&|^~]+ - operator ::= \.+ - - Reserved for punctuation: '.', '=', '->', and unary prefix '&' - Reserved for comments: '//', '/*' and '*/' - - operator-binary ::= operator - operator-prefix ::= operator - operator-postfix ::= operator - - left-binder ::= [ \r\n\t\(\[\{,;:] - right-binder ::= [ \r\n\t\)\]\},;:] - - any-identifier ::= identifier | operator - -``operator-binary``, ``operator-prefix``, and ``operator-postfix`` are -distinguished by immediate lexical context. An operator token is called -*left-bound* if it is immediately preceded by a character matching -``left-binder``. An operator token is called *right-bound* if it is -immediately followed by a character matching ``right-binder``. An operator -token is an ``operator-prefix`` if it is right-bound but not left-bound, an -``operator-postfix`` if it is left-bound but not right-bound, and an -``operator-binary`` in either of the other two cases. - -As an exception, an operator immediately followed by a dot ('``.``') is only -considered right-bound if not already left-bound. This allows ``a!.prop`` to -be parsed as ``(a!).prop`` rather than as ``a ! .prop``. - -The '``!``' operator is postfix if it is left-bound. - -The '``?``' operator is postfix (and therefore not the ternary operator) if it -is left-bound. The sugar form for ``Optional`` types must be left-bound. - -When parsing certain grammatical constructs that involve '``<``' and '``>``' -(such as protocol composition types), an -``operator`` with a leading '``<``' or '``>``' may be split into two or more -tokens: the leading '``<``' or '``>``' and the remainder of the token, which -may be an ``operator`` or ``punctuation`` token that may itself be further -split. This rule allows us to parse nested constructs such as ``A>`` -without requiring spaces between the closing '``>``'s. - -.. _langref.lexical.dollarident: - -Implementation Identifier Token -------------------------------- - -.. code-block:: none - - dollarident ::= '$' id-continue+ - -Tokens that start with a ``$`` are separate class of identifier, which are -fixed purpose names that are defined by the implementation. - -.. _langref.lexical.escapedident: - -Escaped Identifiers -------------------- - -.. code-block:: none - - identifier ::= '`' id-start id-continue* '`' - -An identifier that would normally be a `keyword ` may -be used as an identifier by wrapping it in backticks '``\```', for example:: - - func `class`() { /* ... */ } - let `type` = 0.type - -Any identifier may be escaped, though only identifiers that would normally be -parsed as keywords are required to be. The backtick-quoted string must still -form a valid, non-operator identifier:: - - let `0` = 0 // Error, "0" doesn't start with an alphanumeric - let `foo-bar` = 0 // Error, '-' isn't an identifier character - let `+` = 0 // Error, '+' is an operator - -.. _langref.namebind: - -Name Binding -============ - -.. _langref.typecheck: - -Type Checking -============= - -.. _langref.decl: - -Declarations -============ - -... - -.. _langref.decl.import: - -Import Declarations -------------------- - -... - -.. _langref.decl.var: - -``var`` Declarations --------------------- - -.. code-block:: none - - decl-var-head-kw ::= ('static' | 'class')? 'override'? - decl-var-head-kw ::= 'override'? ('static' | 'class')? - - decl-var-head ::= attribute-list decl-var-head-kw? 'var' - - decl-var ::= decl-var-head pattern initializer? (',' pattern initializer?)* - - // 'get' is implicit in this syntax. - decl-var ::= decl-var-head identifier ':' type brace-item-list - - decl-var ::= decl-var-head identifier ':' type '{' get-set '}' - - decl-var ::= decl-var-head identifier ':' type initializer? '{' willset-didset '}' - - // For use in protocols. - decl-var ::= decl-var-head identifier ':' type '{' get-set-kw '}' - - get-set ::= get set? - get-set ::= set get - - get ::= attribute-list ( 'mutating' | 'nonmutating' )? 'get' brace-item-list - set ::= attribute-list ( 'mutating' | 'nonmutating' )? 'set' set-name? brace-item-list - set-name ::= '(' identifier ')' - - willset-didset ::= willset didset? - willset-didset ::= didset willset? - - willset ::= attribute-list 'willSet' set-name? brace-item-list - didset ::= attribute-list 'didSet' set-name? brace-item-list - - get-kw ::= attribute-list ( 'mutating' | 'nonmutating' )? 'get' - set-kw ::= attribute-list ( 'mutating' | 'nonmutating' )? 'set' - get-set-kw ::= get-kw set-kw? - get-set-kw ::= set-kw get-kw - -``var`` declarations form the backbone of value declarations in Swift. A -``var`` declaration takes a pattern and an optional initializer, and declares -all the pattern-identifiers in the pattern as variables. If there is an -initializer and the pattern is :ref:`fully-typed `, -the initializer is converted to the type of the pattern. If there is an -initializer and the pattern is not fully-typed, the type of initializer is -computed independently of the pattern, and the type of the pattern is derived -from the initializer. If no initializer is specified, the pattern must be -fully-typed, and the values are default-initialized. - -If there is more than one pattern in a ``var`` declaration, they are each -considered independently, as if there were multiple declarations. The initial -``attribute-list`` is shared between all the declared variables. - -A var declaration may contain a getter and (optionally) a setter, which will -be used when reading or writing the variable, respectively. Such a variable -does not have any associated storage. A ``var`` declaration with a getter or -setter must have a type (call it ``T``). The getter function, whose -body is provided as part of the ``var-get`` clause, has type ``() -> T``. -Similarly, the setter function, whose body is part of the ``var-set`` clause -(if provided), has type ``(T) -> ()``. - -If the ``var-set`` or ``willset`` clause contains a ``set-name`` clause, the -identifier of that clause is used as the name of the parameter to the setter or -the observing accessor. Otherwise, the parameter name is ``newValue``. Same -applies to ``didset`` clause, but the default parameter name is ``oldValue``. - -FIXME: Should the type of a pattern which isn't fully typed affect the -type-checking of the expression (i.e. should we compute a structured dependent -type)? - -Like all other declarations, ``var``\ s can optionally have a list of -:ref:`attributes ` applied to them. - -The type of a variable must be :ref:`materializable -`. A variable is an lvalue unless it has a -``var-get`` clause but not ``var-set`` clause. - -Here are some examples of ``var`` declarations: - -:: - - // Simple examples. - var a = 4 - var b: Int - var c: Int = 42 - - // This decodes the tuple return value into independently named parts - // and both 'val' and 'err' are in scope after this line. - var (val, err) = foo() - - // Variable getter/setter - var _x: Int = 0 - var x_modify_count: Int = 0 - var x1: Int { - return _x - } - var x2: Int { - get { - return _x - } - set { - x_modify_count = x_modify_count + 1 - _x = value - } - } - -Note that ``get``, ``set``, ``willSet`` and ``didSet`` are context-sensitive -keywords. - -``static`` keyword is allowed inside structs and enums, and extensions of -those. - -``class`` keyword is allowed inside classes, class extensions, and -protocols. - -.. admonition:: Ambiguity 1 - - The production for implicit ``get`` makes this grammar ambiguous. For example: - - :: - - class A { - func get(_: () -> Int) {} - var a: Int { - get { return 0 } // Getter declaration or call to 'get' with a trailing closure? - } - // But if this was intended as a call to 'get' function, then we have a - // getter without a 'return' statement, so the code is invalid anyway. - } - - We disambiguate towards ``get-set`` or ``willset-didset`` production if the - first token after ``{`` is the corresponding keyword, possibly preceeded by - attributes. Thus, the following code is rejected because we are expecting - ``{`` after ``set``: - - :: - - class A { - var set: Foo - var a: Int { - set.doFoo() - return 0 - } - } - -.. admonition:: Ambiguity 2 - - The production with ``initializer`` and an accessor block is ambiguous. For - example: - - :: - - func takeClosure(_: () -> Int) {} - struct A { - var willSet: Int - var a: Int = takeClosure { - willSet {} // A 'willSet' declaration or a call to 'takeClosure'? - } - } - - We disambiguate towards ``willget-didset`` production if the first token - after ``{`` is the keyword ``willSet`` or ``didSet``, possibly preceeded by - attributes. - -.. admonition:: Rationale - - Even though it is possible to do further checks and speculatively parse more, - it introduces unjustified complexity to cover (hopefully rare) corner cases. - In ambiguous cases users can always opt-out of the trailing closure syntax by - using explicit parentheses in the function call. - -.. _langref.decl.func: - -``func`` Declarations ---------------------- - -.. code-block:: none - - // Keywords can be specified in any order. - decl-func-head-kw ::= ( 'static' | 'class' )? 'override'? ( 'mutating' | 'nonmutating' )? - - decl-func ::= attribute-list decl-func-head-kw? 'func' any-identifier generic-params? func-signature brace-item-list? - -``func`` is a declaration for a function. The argument list and optional -return value are specified by the type production of the function, and the body -is either a brace expression or elided. Like all other declarations, functions -are can have attributes. - -If the type is not syntactically a function type (i.e., has no ``->`` in it at -top-level), then the return value is implicitly inferred to be ``()``. All of -the argument and return value names are injected into the scope of the function body.

- -A function in an extension of some type (or -in other places that are semantically equivalent to an extension) implicitly -get a ``self`` argument with these rules ... [todo] - -``static`` and ``class`` functions are only allowed in an extension of some type (or in other places that are -semantically equivalent to an extension). They indicate that the function is -actually defined on the metatype for the type, not on -the type itself. Thus its implicit ``self`` argument is actually of -metatype type. - -``static`` keyword is allowed inside structs and enums, and extensions of those. - -``class`` keyword is allowed inside classes, class extensions, and protocols. - -TODO: Func should be an immutable name binding, it should implicitly add an -attribute immutable when it exists. - -TODO: Incoming arguments should be readonly, result should be implicitly -writeonly when we have these attributes. - -.. _langref.decl.func.signature: - -Function signatures -^^^^^^^^^^^^^^^^^^^ - -... - -An argument name is a keyword argument if: -- It is an argument to an initializer, or -- It is an argument to a method after the first argument, or -- It is preceded by a back-tick (`), or -- Both a keyword argument name and an internal parameter name are specified. - -.. _langref.decl.subscript: - -``subscript`` Declarations ---------------------------- - -.. code-block:: none - - subscript-head ::= attribute-list 'override'? 'subscript' pattern-tuple '->' type - - decl-subscript ::= subscript-head '{' get-set '}' - - // 'get' is implicit in this syntax. - decl-subscript ::= subscript-head brace-item-list - - // For use in protocols. - decl-subscript ::= subscript-head '{' get-set-kw '}' - -A subscript declaration provides support for -subscripting an object of a particular type via a getter and (optional) -setter. Therefore, subscript declarations can only appear within a type -definition or extension. - -The ``pattern-tuple`` of a subscript declaration provides the indices that -will be used in the subscript expression, e.g., the ``i`` in ``a[i]``. This -pattern must be fully-typed. The ``type`` following the arrow provides the -type of element being accessed, which must be materializable. Subscript -declarations can be overloaded, so long as either the ``pattern-tuple`` or -``type`` differs from other declarations. - -The ``get-set`` clause specifies the getter and setter used for subscripting. -The getter is a function whose input is the type of the ``pattern-tuple`` and -whose result is the element type. Similarly, the setter is a function whose -result type is ``()`` and whose input is the type of the ``pattern-tuple`` -with a parameter of the element type added to the end of the tuple; the name -of the parameter is the ``set-name``, if provided, or ``value`` otherwise. - -:: - - // Simple bit vector with storage for 64 boolean values - struct BitVector64 { - var bits: Int64 - - // Allow subscripting with integer subscripts and a boolean result. - subscript (bit : Int) -> Bool { - // Getter tests the given bit - get { - return bits & (1 << bit)) != 0 - } - - // Setter sets the given bit to the provided value. - set { - var mask = 1 << bit - if value { - bits = bits | mask - } else { - bits = bits & ~mask - } - } - } - } - - var vec = BitVector64() - vec[2] = true - if vec[3] { - print("third bit is set") - } - -.. _langref.decl.attribute_list: - -Attribute Lists ---------------- - -... - -.. _langref.types: - -Types -===== - -... - -.. _langref.types.fully_typed: - -Fully-Typed Types ------------------ - -... - -.. _langref.types.materializable: - -Materializable Types --------------------- - -... - -.. _langref.pattern: - -Patterns -======== - -.. admonition:: Commentary - - The pattern grammar mirrors the expression grammar, or to be more specific, - the grammar of literals. This is because the conceptual algorithm for - matching a value against a pattern is to try to find an assignment of values - to variables which makes the pattern equal the value. So every expression - form which can be used to build a value directly should generally have a - corresponding pattern form. - -.. code-block:: none - - pattern-atom ::= pattern-var - pattern-atom ::= pattern-any - pattern-atom ::= pattern-tuple - pattern-atom ::= pattern-is - pattern-atom ::= pattern-enum-element - pattern-atom ::= expr - - pattern ::= pattern-atom - pattern ::= pattern-typed - -A pattern represents the structure of a composite value. Parts of a value can -be extracted and bound to variables or compared against other values by -*pattern matching*. Among other places, pattern matching occurs on the -left-hand side of :ref:`var bindings `, in the arguments of -:ref:`func declarations `, and in the case labels -of :ref:`switch statements `. Some examples:: - - var point = (1, 0, 0) - - // Extract the elements of the "point" tuple and bind them to - // variables x, y, and z. - var (x, y, z) = point - print("x=\(x) y=\(y) z=\(z)") - - // Dispatch on the elements of a tuple in a "switch" statement. - switch point { - case (0, 0, 0): - print("origin") - // The pattern "_" matches any value. - case (_, 0, 0): - print("on the x axis") - case (0, _, 0): - print("on the y axis") - case (0, 0, _): - print("on the z axis") - case (var x, var y, var z): - print("x=\(x) y=\(y) z=\(z)") - } - - -A pattern may be "irrefutable", meaning informally that it matches all values -of its type. Patterns in declarations, such as :ref:`var ` -and :ref:`func `, are required to be irrefutable. Patterns -in the ``case`` labels of :ref:`switch statements `, -however, are not. - -The basic pattern grammar is a literal "atom" followed by an optional type -annotation. Type annotations are useful for documentation, as well as for -coercing a matched expression to a particular kind. They are also required -when patterns are used in a :ref:`function signature -`. Type annotations are currently not allowed in -switch statements. - -A pattern has a type. A pattern may be "fully-typed", meaning informally that -its type is fully determined by the type annotations it contains. Some -patterns may also derive a type from their context, be it an enclosing pattern -or the way it is used; this set of situations is not yet fully determined. - -.. _langref.pattern.typed: - -Typed Patterns --------------- - -.. code-block:: none - - pattern-typed ::= pattern-atom ':' type - -A type annotation constrains a pattern to have a specific type. An annotated -pattern is fully-typed if its annotation type is fully-typed. It is -irrefutable if and only if its subpattern is irrefutable. - -Type annotations are currently not allowed in the ``case`` labels of ``switch`` -statements; case patterns always get their type from the subject of the switch. - -.. _langref.pattern.any: - -Any Patterns ------------- - -.. code-block:: none - - pattern-any ::= '_' - -The symbol ``_`` in a pattern matches and ignores any value. It is irrefutable. - -.. _langref.pattern.var: - -'var' and 'let' Patterns ------------------------- - -.. code-block:: none - - pattern-var ::= 'let' pattern - pattern-var ::= 'var' pattern - -The ``var`` and ``let`` keywords within a pattern introduces variable bindings. -Any identifiers within the subpattern bind new named variables to their -matching values. 'var' bindings are mutable within the bound scope, and 'let' -bindings are immutable. - -:: - - var point = (0, 0, 0) - switch point { - // Bind x, y, z to the elements of point. - case (var x, var y, var z): - print("x=\(x) y=\(y) z=\(z)") - } - - switch point { - // Same. 'var' distributes to the identifiers in its subpattern. - case var (x, y, z): - print("x=\(x) y=\(y) z=\(z)") - } - -Outside of a var pattern, an identifier behaves as an :ref:`expression -pattern ` referencing an existing definition. - -:: - - var zero = 0 - switch point { - // x and z are bound as new variables. - // zero is a reference to the existing 'zero' variable. - case (var x, zero, var z): - print("point off the y axis: x=\(x) z=\(z)") - default: - print("on the y axis") - } - -The left-hand pattern of a :ref:`var declaration ` and the -argument pattern of a :ref:`func declaration ` are -implicitly inside a ``var`` pattern; identifiers in their patterns always bind -variables. Variable bindings are irrefutable. - -The type of a bound variable must be :ref:`materializable -` unless it appears in a :ref:`func-signature -` and is directly of a ``inout``\ -annotated type. - -.. _langref.pattern.tuple: - -Tuple Patterns --------------- - -.. code-block:: none - - pattern-tuple ::= '(' pattern-tuple-body? ')' - pattern-tuple-body ::= pattern-tuple-element (',' pattern-tuple-body)* '...'? - pattern-tuple-element ::= pattern - -A tuple pattern is a list of zero or more patterns. Within a :ref:`function -signature `, patterns may also be given a -default-value expression. - -A tuple pattern is irrefutable if all its sub-patterns are irrefutable. - -A tuple pattern is fully-typed if all its sub-patterns are fully-typed, in -which case its type is the corresponding tuple type, where each -``type-tuple-element`` has the type, label, and default value of the -corresponding ``pattern-tuple-element``. A ``pattern-tuple-element`` has a -label if it is a named pattern or a type annotation of a named pattern. - -A tuple pattern whose body ends in ``'...'`` is a varargs tuple. The last -element of such a tuple must be a typed pattern, and the type of that pattern -is changed from ``T`` to ``T[]``. The corresponding tuple type for a varargs -tuple is a varargs tuple type. - -As a special case, a tuple pattern with one element that has no label, has no -default value, and is not varargs is treated as a grouping parenthesis: it has -the type of its constituent pattern, not a tuple type. - -.. _langref.pattern.is: - -``is`` Patterns ---------------- - -.. code-block:: none - - pattern-is ::= 'is' type - -``is`` patterns perform a type check equivalent to the ``x is T`` cast operator. The pattern matches if the runtime type of -a value is of the given type. ``is`` patterns are refutable and thus cannot -appear in declarations. - -:: - - class B {} - class D1 : B {} - class D2 : B {} - - var bs : B[] = [B(), D1(), D2()] - - for b in bs { - switch b { - case is B: - print("B") - case is D1: - print("D1") - case is D2: - print("D2") - } - } - - -.. _langref.pattern.enum_element: - -Enum Element Patterns ---------------------- - -.. code-block:: none - - pattern-enum-element ::= type-identifier? '.' identifier pattern-tuple? - -Enum element patterns match a value of enum type if -the value matches the referenced ``case`` of the enum. If the ``case`` has a -type, the value of that type can be matched against an optional subpattern. - -:: - - enum HTMLTag { - case A(href: String) - case IMG(src: String, alt: String) - case BR - } - - switch tag { - case .BR: - print("
") - case .IMG(var src, var alt): - print("\"\(escape(alt))\"") - case .A(var href): - print("") - } - -Enum element patterns are refutable and thus cannot appear in declarations. -(They are currently considered refutable even if the enum contains only a -single ``case``.) - -.. _langref.pattern.expr: - -Expressions in Patterns ------------------------ - -Patterns may include arbitrary expressions as subpatterns. Expression patterns -are refutable and thus cannot appear in declarations. An expression pattern is -compared to its corresponding value using the ``~=`` operator. The match -succeeds if ``expr ~= value`` evaluates to true. The standard library provides -a default implementation of ``~=`` using ``==`` equality; additionally, range -objects may be matched against integer and floating-point values. The ``~=`` -operator may be overloaded like any function. - -:: - - var point = (0, 0, 0) - switch point { - // Equality comparison. - case (0, 0, 0): - print("origin") - // Range comparison. - case (-10...10, -10...10, -10...10): - print("close to the origin") - default: - print("too far away") - } - - // Define pattern matching of an integer value to a string expression. - func ~=(pattern:String, value:Int) -> Bool { - return pattern == "\(value)" - } - - // Now we can pattern-match strings to integers: - switch point { - case ("0", "0", "0"): - print("origin") - default: - print("not the origin") - } - -The order of evaluation of expressions in patterns, including whether an -expression is evaluated at all, is unspecified. The compiler is free to -reorder or elide expression evaluation in patterns to improve dispatch -efficiency. Expressions in patterns therefore cannot be relied on for side -effects. - -.. _langref.expr: - -Expressions -=========== - -... - -.. _langref.expr.call: - -Function Application --------------------- - -... - -.. _langref.stmt: - -Statements -========== - -... - -.. _langref.stmt.break: - -``break`` Statement -------------------- - -.. code-block:: none - - stmt-return ::= 'break' - -The 'break' statement transfers control out of the enclosing 'for' loop or -'while' loop. - -.. _langref.stmt.continue: - -``continue`` Statement ----------------------- - -.. code-block:: none - - stmt-return ::= 'continue' - -The 'continue' statement transfers control back to the start of the enclosing -'for' loop or 'while' loop. - -... - -.. _langref.stmt.switch: - -``switch`` Statement --------------------- - -.. code-block:: none - - stmt-switch ::= 'switch' expr-basic '{' stmt-switch-case* '}' - - stmt-switch-case ::= (case-label | default-label) brace-item+ - stmt-switch-case ::= (case-label | default-label) ';' - - case-label ::= 'case' pattern ('where' expr)? (',' pattern ('where' expr)?)* ':' - default-label ::= 'default' ':' - -'switch' statements branch on the value of an expression by :ref:`pattern -matching `. The subject expression of the switch is evaluated -and tested against the patterns in its ``case`` labels in source order. When a -pattern is found that matches the value, control is transferred into the -matching ``case`` block. ``case`` labels may declare multiple patterns -separated by commas. Only a single ``case`` labels may precede a block of -code. Case labels may optionally specify a *guard* expression, introduced by -the ``where`` keyword; if present, control is transferred to the case only if -the subject value both matches the corresponding pattern and the guard -expression evaluates to true. Patterns are tested "as if" in source order; if -multiple cases can match a value, control is transferred only to the first -matching case. The actual execution order of pattern matching operations, and -in particular the evaluation order of :ref:`expression patterns -`, is unspecified. - -A switch may also contain a ``default`` block. If present, it receives control -if no cases match the subject value. The ``default`` block must appear at the -end of the switch and must be the only label for its block. ``default`` is -equivalent to a final ``case _`` pattern. Switches are required to be -exhaustive; either the contained case patterns must cover every possible value -of the subject's type, or else an explicit ``default`` block must be specified -to handle uncovered cases. - -Every case and default block has its own scope. Declarations within a case or -default block are only visible within that block. Case patterns may bind -variables using the :ref:`var keyword `; those variables -are also scoped into the corresponding case block, and may be referenced in the -``where`` guard for the case label. However, if a case block matches multiple -patterns, none of those patterns may contain variable bindings. - -Control does not implicitly 'fall through' from one case block to the next. -:ref:`fallthrough statements ` may explicitly -transfer control among case blocks. :ref:`break ` and -:ref:`continue ` within a switch will break or continue -out of an enclosing 'while' or 'for' loop, not out of the 'switch' itself. - -At least one ``brace-item`` is required in every case or default block. It is -allowed to be a no-op. Semicolon can be used as a single no-op statement in -otherwise empty cases in switch statements. - -:: - - func classifyPoint(point: (Int, Int)) { - switch point { - case (0, 0): - print("origin") - - case (_, 0): - print("on the x axis") - - case (0, _): - print("on the y axis") - - case (var x, var y) where x == y: - print("on the y = x diagonal") - - case (var x, var y) where -x == y: - print("on the y = -x diagonal") - - case (var x, var y): - print("length \(sqrt(x*x + y*y))") - } - } - - switch x { - case 1, 2, 3: - print("x is 1, 2 or 3") - default: - ; - } - -.. _langref.stmt.fallthrough: - -``fallthrough`` Statement -------------------------- - -.. code-block:: none - - stmt-fallthrough ::= 'fallthrough' - -``fallthrough`` transfers control from a ``case`` block of a :ref:`switch -statement ` to the next ``case`` or ``default`` block -within the switch. It may only appear inside a ``switch``. ``fallthrough`` -cannot be used in the final block of a ``switch``. It also cannot transfer -control into a ``case`` block whose pattern contains :ref:`var bindings -`. - -.. _langref.stdlib: - -Standard Library -================ - -.. admonition:: Commentary - - It would be really great to have literate swift code someday, that way this - could be generated directly from the code. This would also be powerful for - Swift library developers to be able to depend on being available and - standardized. - -This describes some of the standard swift code as it is being built up. Since -Swift is designed to give power to the library developers, much of what is -normally considered the "language" is actually just implemented in the -library. - -All of this code is published by the '``swift``' module, which is implicitly -imported into each source file, unless some sort of pragma in the code -(attribute on an import?) is used to change or disable this behavior. - -.. _langref.stdlib.builtin: - -Builtin Module -============== - -In the initial Swift implementation, a module named ``Builtin`` is imported -into every file. Its declarations can only be found by dot -syntax. It provides access to a small number of primitive representation -types and operations defined over them that map directly to LLVM IR. - -The existence of and details of this module are a private implementation detail -used by our implementation of the standard library. Swift code outside the -standard library should not be aware of this library, and an independent -implementation of the swift standard library should be allowed to be -implemented without the builtin library if it desires. - -For reference below, the description of the standard library uses the -"``Builtin.``" namespace to refer to this module, but independent -implementations could use another implementation if they so desire. - -.. _langref.stdlib.simple-types: - -Simple Types ------------- - -Void -^^^^ - -:: - - // Void is just a type alias for the empty tuple. - typealias Void = () - -.. _langref.stdlib.int: - -Int, Int8, Int16, Int32, Int64 -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -.. admonition:: Commentary - - Having a single standardized integer type that can be used by default - everywhere is important. One advantage Swift has is that by the time it is - in widespread use, 64-bit architectures will be pervasive, and the LLVM - optimizer should grow to be good at shrinking 64-bit integers to 32-bit in - many cases for those 32-bit architectures that persist. - -:: - - // Fixed size types are simple structs of the right size. - struct Int8 { value : Builtin.Int8 } - struct Int16 { value : Builtin.Int16 } - struct Int32 { value : Builtin.Int32 } - struct Int64 { value : Builtin.Int64 } - struct Int128 { value : Builtin.Int128 } - - // Int is just an alias for the 64-bit integer type. - typealias Int = Int64 - -.. _langref.stdlib.float: - -Int, Int8, Int16, Int32, Int64 -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -:: - - struct Float { value : Builtin.FPIEEE32 } - struct Double { value : Builtin.FPIEEE64 } - -.. _langref.stdlib.bool: - -Bool, true, false -^^^^^^^^^^^^^^^^^ - -:: - - // Bool is a simple enum. - enum Bool { - true, false - } - - // Allow true and false to be used unqualified. - var true = Bool.true - var false = Bool.false - -.. _langref.stdlib.oper: - -Arithmetic and Logical Operations ---------------------------------- - -.. _langref.stdlib.oper.arithmetic: - -Arithmetic Operators -^^^^^^^^^^^^^^^^^^^^ - -:: - - func * (lhs: Int, rhs: Int) -> Int - func / (lhs: Int, rhs: Int) -> Int - func % (lhs: Int, rhs: Int) -> Int - func + (lhs: Int, rhs: Int) -> Int - func - (lhs: Int, rhs: Int) -> Int - -.. _langref.stdlib.oper.comparison: - -Relational and Equality Operators -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -:: - - func < (lhs : Int, rhs : Int) -> Bool - func > (lhs : Int, rhs : Int) -> Bool - func <= (lhs : Int, rhs : Int) -> Bool - func >= (lhs : Int, rhs : Int) -> Bool - func == (lhs : Int, rhs : Int) -> Bool - func != (lhs : Int, rhs : Int) -> Bool - -.. _langref.stdlib.oper.short-circuit-logical: - -Short Circuiting Logical Operators -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -:: - - func && (lhs: Bool, rhs: ()->Bool) -> Bool - func || (lhs: Bool, rhs: ()->Bool) -> Bool - -Swift has a simplified precedence levels when compared with C. From highest to -lowest: - -:: - - "exponentiative:" <<, >> - "multiplicative:" *, /, %, & - "additive:" +, -, |, ^ - "comparative:" ==, !=, <, <=, >=, > - "conjunctive:" && - "disjunctive:" || - - diff --git a/docs/archive/Namespace Level Vars and Top Level Code.md b/docs/archive/Namespace Level Vars and Top Level Code.md new file mode 100644 index 0000000000000..ae2c53244be22 --- /dev/null +++ b/docs/archive/Namespace Level Vars and Top Level Code.md @@ -0,0 +1,134 @@ +> **warning** +> +> This document was used in planning Swift 1.0; it has not been kept + +> up to date and does not describe the current or planned behavior of +> Swift. + +Mutable Namespace-Scope Variable Declarations +============================================= + +A namespace-scope variable (i.e. a variable not inside a function) is +allowed to have an initializer, and that initializer is allowed to have +side effects. Thus, we have to decide how and when the initializer runs. + +WLOG, lets assume that all namespace-scope variables are mutable (and +thus that immutable variables are just an optimization of the common +case). Given that they can have mutable state, they cannot be "global" +(in the C sense) because then they would be visible across multiple +actors. Instead, the only logical semantic is for them to be actor-local +data ("thread local" in the C sense) of some sort. + +Given that there can be many of these variables in an address space, and +very few of them may be dynamically used by any particular actor, it +doesn't make sense to allocate space for all of the variables and run +all of the initializers for the variables at actor-startup-time. +Instead, swift should handle these as "actor associated data" (stored in +a hashtable that the actor has a pointer to) and should be lazily +initialized (in the absence of 'top level code', see below). + +This means that if you write code like this (optionally we could require +an attribute to make it clear that the value is actor local): + + func foo(a : int) -> int { print(a) return 0 } + + var x = foo(1) + var y = foo(2) + +That the print statements will execute the first time that x or y is +used by any particular actor. + +Top Level Code +-------------- + +One goal of swift is to provide a very "progressive disclosure" model of +writing code and learning how to write code. Therefore, it is desirable +that someone be able to start out with: + + print("hello world\n") + +as their first application. This requires that we support "top level +code", which is code outside any function or other declaration. The +counter-example of this is java, which requires someone to look at +"class foo / public static void main String\[\]...." all of which is +non-essential to the problem of writing a simple app. + +Top level code is useful for a number of other things: many scripts +written by unix hackers (in perl, bourne shell, ruby, etc) are really +just simple command line apps that may have a few helper functions and +some code that runs. While not essential, it is a great secondary goal +to make these sorts of simple apps easy to write in Swift as well. + +Top-Level code and lazily evaluated variable initializers don't mix +well, nor does top level code and multiple actors. As such, the logical +semantics are: + +1. Source files are partitioned into two cases: "has TLC" and "has + no TLC". +2. All variables defined in "has no TLC" files are allocated and + initialized lazily. +3. Source files that have TLC are each initialized in a deterministic + order: The dependence graph of domains is respected (lower level + domains are initialized before dependent ones), and the source files + withing a domain are initialized in some deterministic order + (perhaps according to their filename or something, TBD). +4. Within a source file with TLC, the TLC is run top down in + determinstic order whenever the file's initializer is run. This + initializer executes in the context of the "first" actor, which is + created on behalf of the program by the runtime library. +5. If/when some other actor refers to a variable in a file with TLC, it + is allocated and initialized lazily just like globals in "has no + TLC" files. + +On "Not Having Headers" +----------------------- + +One intentional design decision in swift is to not have header files, +even for public API. This is a design point like Java, but unlike C or +Objective-C. Having header files for public API is nice for a couple of +reasons: + +1. You *force* people to think about what they are making public, and + the act of having to put it into a header makes them think about its + fragility and duration over time. +2. Headers are a very convenient place to get an overview of what an + API is and does. In Java you get all the implementation details of a + class mixed in with its public API, which makes it very difficult to + understand "at a glance" what a class does. +3. Headers are very useful documentation for Objective-C because we + ship the headers but not the implementation of system classes. This + allows "jump to definition" to go to the declaration of an API in + the header, which is conveniently co-located with headerdoc. + +On the other hand, headers have a number of disadvantages including: + +1. It is plain code duplication, with all the negative effects of it. + It slows down development, can get out of synch, makes changing API + more difficult, etc. +2. If the prototype and implementation get out of synch, it is caught + by the compiler, but this isn't true for API comments. +3. Swift natively won't "need" headers, so we'd have to specifically + add this capability, making the language more complicated. +4. The implementation of a framework may not be in swift. If you're + talking to a C or C++ framework, showing a C or C++ header when + "jumping to definition" is not particularly helpful. We'd prefer to + show you the synthesized API that swift code should be using. +5. In Swift, the implementation of some datatype can be split across + different files. Forcing all their declarations to be next to each + other lexically is an arbitrary restriction. + +To address the disadvantages of not having headers, we think that we +should: + +1. Standardize on a syntax for doc comments, and bake it into + the language. Mistakes using it should be diagnosed by the compiler. + It should be a warning for public API to not have comments. +2. There needs to be an API that dumps out the public interface for a + compiled module/domain in swift syntax, slicing on a declaration. + When used on a type, for example, this would show the type + definition and the declaration of all of the methods on it. +3. The API dumper should always dump in swift syntax, even when run on + a Clang C/C++/ObjC module. It should make it very clear what the API + maps to in swift syntax, so it is obvious how to use it. +4. Not having headers forces us to have really great + tools support/integration. diff --git a/docs/archive/Namespace Level Vars and Top Level Code.rst b/docs/archive/Namespace Level Vars and Top Level Code.rst deleted file mode 100644 index ba140a8286acd..0000000000000 --- a/docs/archive/Namespace Level Vars and Top Level Code.rst +++ /dev/null @@ -1,129 +0,0 @@ -:orphan: - -.. @raise litre.TestsAreMissing - -.. warning:: This document was used in planning Swift 1.0; it has not been kept - up to date and does not describe the current or planned behavior of Swift. - -Mutable Namespace-Scope Variable Declarations -============================================= - -A namespace-scope variable (i.e. a variable not inside a function) is allowed to -have an initializer, and that initializer is allowed to have side effects. -Thus, we have to decide how and when the initializer runs. - -WLOG, lets assume that all namespace-scope variables are mutable (and thus that -immutable variables are just an optimization of the common case). Given that -they can have mutable state, they cannot be "global" (in the C sense) because -then they would be visible across multiple actors. Instead, the only logical -semantic is for them to be actor-local data ("thread local" in the C sense) of -some sort. - -Given that there can be many of these variables in an address space, and very -few of them may be dynamically used by any particular actor, it doesn't make -sense to allocate space for all of the variables and run all of the initializers -for the variables at actor-startup-time. Instead, swift should handle these as -"actor associated data" (stored in a hashtable that the actor has a pointer to) -and should be lazily initialized (in the absence of 'top level code', see -below). - -This means that if you write code like this (optionally we could require an -attribute to make it clear that the value is actor local):: - - func foo(a : int) -> int { print(a) return 0 } - - var x = foo(1) - var y = foo(2) - -That the print statements will execute the first time that x or y is used by any -particular actor. - - -Top Level Code --------------- - -One goal of swift is to provide a very "progressive disclosure" model of writing -code and learning how to write code. Therefore, it is desirable that someone be -able to start out with:: - - print("hello world\n") - -as their first application. This requires that we support "top level code", -which is code outside any function or other declaration. The counter-example of -this is java, which requires someone to look at "class foo / public static void -main String[]...." all of which is non-essential to the problem of writing a -simple app. - -Top level code is useful for a number of other things: many scripts written by -unix hackers (in perl, bourne shell, ruby, etc) are really just simple command -line apps that may have a few helper functions and some code that runs. While -not essential, it is a great secondary goal to make these sorts of simple apps -easy to write in Swift as well. - -Top-Level code and lazily evaluated variable initializers don't mix well, nor -does top level code and multiple actors. As such, the logical semantics are: - -1. Source files are partitioned into two cases: "has TLC" and "has no TLC". -2. All variables defined in "has no TLC" files are allocated and initialized - lazily. -3. Source files that have TLC are each initialized in a deterministic order: The - dependence graph of domains is respected (lower level domains are initialized - before dependent ones), and the source files withing a domain are initialized - in some deterministic order (perhaps according to their filename or - something, TBD). -4. Within a source file with TLC, the TLC is run top down in determinstic order - whenever the file's initializer is run. This initializer executes in the - context of the "first" actor, which is created on behalf of the program by - the runtime library. -5. If/when some other actor refers to a variable in a file with TLC, it is - allocated and initialized lazily just like globals in "has no TLC" files. - -On "Not Having Headers" ------------------------ - -One intentional design decision in swift is to not have header files, even for -public API. This is a design point like Java, but unlike C or Objective-C. -Having header files for public API is nice for a couple of reasons: - -1. You *force* people to think about what they are making public, and the act of - having to put it into a header makes them think about its fragility and - duration over time. -2. Headers are a very convenient place to get an overview of what an API is and - does. In Java you get all the implementation details of a class mixed in - with its public API, which makes it very difficult to understand "at a - glance" what a class does. -3. Headers are very useful documentation for Objective-C because we ship the - headers but not the implementation of system classes. This allows "jump to - definition" to go to the declaration of an API in the header, which is - conveniently co-located with headerdoc. - -On the other hand, headers have a number of disadvantages including: - -1. It is plain code duplication, with all the negative effects of it. It slows - down development, can get out of synch, makes changing API more difficult, - etc. -2. If the prototype and implementation get out of synch, it is caught by the - compiler, but this isn't true for API comments. -3. Swift natively won't "need" headers, so we'd have to specifically add this - capability, making the language more complicated. -4. The implementation of a framework may not be in swift. If you're talking to - a C or C++ framework, showing a C or C++ header when "jumping to definition" - is not particularly helpful. We'd prefer to show you the synthesized API - that swift code should be using. -5. In Swift, the implementation of some datatype can be split across different - files. Forcing all their declarations to be next to each other lexically is - an arbitrary restriction. - -To address the disadvantages of not having headers, we think that we should: - -1. Standardize on a syntax for doc comments, and bake it into the language. - Mistakes using it should be diagnosed by the compiler. It should be a - warning for public API to not have comments. -2. There needs to be an API that dumps out the public interface for a compiled - module/domain in swift syntax, slicing on a declaration. When used on a - type, for example, this would show the type definition and the declaration of - all of the methods on it. -3. The API dumper should always dump in swift syntax, even when run on a Clang - C/C++/ObjC module. It should make it very clear what the API maps to in - swift syntax, so it is obvious how to use it. -4. Not having headers forces us to have really great tools support/integration. diff --git a/docs/archive/Objective-C Interoperability.md b/docs/archive/Objective-C Interoperability.md new file mode 100644 index 0000000000000..bd644222e6b43 --- /dev/null +++ b/docs/archive/Objective-C Interoperability.md @@ -0,0 +1,641 @@ +Objective-C Interoperability +============================ + +This document tracks the differences between the Swift and Objective-C +ABIs and class models, and what it would take to merge the two as much +as possible. The format of each section lays out the differences between +Swift and Objective-C, then describes what needs to happen for a user to +mix the two seamlessly. + +> **warning** +> +> This document was used in planning Swift 1.0; it has not been kept + +> up to date and does not describe the current or planned behavior of +> Swift. + +Terminology used in this document: + +- `id`-compatible: something that can be assigned to an `id` variable + and sent messages using `objc_msgSend`. In practice, this probably + means implementing the `NSObject` protocol, since most of Cocoa + doesn't check whether something implements `NSObject` before sending + a message like `-class`. +- Objective-C isa: something that identifies the class of an + Objective-C object, used by `objc_msgSend`. To say a Swift object + has an Objective-C isa does *not* mean that a fully-formed + Objective-C runtime class structure is generated for the Swift + class; it just means that (1) the header of the Swift object "looks + like" an Objective-C object, and (2) the parts of an Objective-C + class used by the `objc_msgSend` "fast path" are the same. + +Design +------ + +All Swift objects [^1] will be `id`-compatible and will have an +Objective-C isa, on the assumption that you want to be able to put them +in an array, set them as represented objects, etc. [^2] + +Swift classes that inherit from NSObject (directly or indirectly [^3]) +behave exactly like Objective-C classes from the perspective of +Objective-C source. All methods marked as "API" in Swift will have dual +entry points exposed by default. Methods not marked as "API" will not be +exposed to Objective-C at all. Instances of these classes can be used +like any other Objective-C objects. + +Subclassing a "Swift NSObject class" in Objective-C requires a bit of +extra work: generating Swift vtables. We haven't decided how to do this: + +- Clang could be taught about Swift class layout. +- The Clang driver could call out to the Swift compiler to do this. + Somehow. +- The runtime could fill in the vtable from the Objective-C isa list + at class load time. (This could be necessary anyway to support + dynamic subclassing... which we may or may not do.) + +Swift classes that do not inherit from NSObject are not visible from +Objective-C. Their instances can be manipulated as `id`, or via whatever +protocols they may implement. + + class AppController : NSApplicationDelegate { + func applicationDidFinishLaunching(notification : NSNotification) { + // do stuff + } + } + + // Use 'id ' in Objective-C. + +Like "Swift NSObject classes", though, "pure" Swift classes will still +have an isa, and any methods declared in an Objective-C protocol will be +emitted with dual entry points. + +Use Cases +--------- + +*Unfinished and undetailed.* + +### Simple Application Writer + +I want to write my new iOS application in Swift, using all the +Objective-C frameworks that come with iOS. + +Guidelines: + +Everything should Just Work™. There should be no need to subclass +NSObject anywhere in your program, unless you are specifically +specializing a class in the Cocoa Touch frameworks. + +### Intermediate Application Writer + +I want to write my new application in Objective-C, but there's a really +nice Swift framework I want to use. + +Guidelines: + +- Not all Swift methods in the framework may be available + in Objective-C. You can work around this by adding *extensions* to + the Swift framework classes to expose a more + Objective-C-friendly interface. You will need to mark these new + methods as "API" in order to make them visible to Objective-C. +- "Pure" Swift classes will not be visible to Objective-C at all. You + will have to write a wrapper class (or wrapper functions) in Swift + if you want to use the features of these classes directly. However, + you can still treat them like any other objects in your program + (store them in `id` variables, Objective-C collections, etc). + +### Transitioning Application Writer + +I have an existing Objective-C application, and I want to convert it +piece-by-piece to Swift. + +Guidelines: + +- Swift is different from Objective-C in that methods in Swift classes + are not automatically usable from everywhere. If your Swift class + inherits from NSObject, marking your methods as "API" will allow + them to be called from Objective-C code. A Swift class that does not + inherit from NSObject will only respond to messages included in its + adopted protocols. [^4] +- Once you have finished transitioning to Swift, go through your + classes and remove the "API" marker from any methods that do not + need to be accessed from Objective-C. Remove NSObject as a + superclass from any classes that do not need to be accessed + from Objective-C. Both of these allow the compiler to be more + aggressive in optimizing your program, potentially making it both + smaller and faster. + +### New Framework Writer + +I want to write a framework that can be used by anyone. + +Requirements: + +- Can call (at least some) Swift methods from Objective-C. + +### Intermediate Framework Writer + +I have an existing Objective-C framework that I want to move to Swift. + +Requirements: + +- Can subclass Objective-C classes in Swift. +- Can call (at least some) Swift methods from Objective-C. + +Decisions: + +- Should I expose Swift entry points as API? +- If so, should they be essentially the same as the Objective-C entry + points, or should I have a very different interface that's more + suited for Swift (and easily could be "better")? + +### End User + +- Things should be fast. +- Things should not take a ton of memory. + +### Nice to Have (uncategorized) + +- Can write a Swift extension for an Objective-C class. +- Can write a Swift extension for an Objective-C class that adopts an + Objective-C protocol. +- Can write a Swift extension for an Objective-C class that exposes + arbitrary new methods in Objective-C. + +Tradeoffs +--------- + +This section discusses models for various runtime data structures, and +the tradeoffs for making Swift's models different from Objective-C. + +### Messaging Model + +Everything is `id`-compatible: + +- Less to think about, maximum compatibility. +- Every Swift object must have an Objective-C isa. + +Non-NSObjects are messageable but not `id`-compatible: + +- Cannot assign Swift objects to `id` variables. +- Cannot put arbitrary Swift objects in NSArrays. +- Potentially confusing: "I can message it but I can't put it in an + `id`??" +- Clang must be taught how to message Swift objects and manage their + retain counts. +- On the plus side, then non-NSObjects can use Swift + calling conventions. +- Requires framework authors to make an arbitrary decision that may + not be ABI-future-proof. + +Non-NSObjects are opaque: + +- Can be passed around, but not manipulated. +- ...but Clang probably *still* has to be taught how to manage the + retain count of an opaque Swift object, and doing so in the same way + as dispatch\_queue\_t and friends may be dangerous (see + <os/object.h> -- it's pretending they're NSObjects, which + they are) +- Requires framework authors to make an arbitrary decision that may + not be ABI-future-proof. + +### Method Model + +*This only affects methods marked as "API" in some way. Assume for now +that all methods use types shared by both Objective-C and Swift, and +that calls within the module can still be optimized away. Therefore, +this discussion only applies to frameworks, and specifically the use of +Swift methods from outside of the module in which they are defined.* + +Every method marked as API can *only* be accessed via Objective-C entry +points: + +- Less to think about, maximum compatibility. +- Penalizes future Swift clients (and potentially + Objective-C clients?). + +Every method marked as API can be accessed both from Objective-C and +Swift: + +- Maximum potential performance. +- Increases binary size and linking time. +- If this is a framework converted to Swift, clients that link against + the Swift entry points are no longer backwards-compatible. And it's + hard to know what you did wrong here. +- Overriding the method in Objective-C requires teaching Clang to emit + a Swift vtable for the subclass. + +Methods marked as "ObjC API" can only be accessed via Objective-C entry +points; methods marked as "Swift API" can only be accessed via Swift +entry points: + +- Changing the API mode breaks binary compatibility. +- Obviously this attribute is inherited -- overriding an Objective-C + method should produce a new Objective-C entry point. What is the + default for new methods, though? Always Swift? Always Objective-C? + Based on the class model (see below)? Specified manually? + +Methods marked as "ObjC API" can be accessed both from Objective-C and +Swift; methods marked as "Swift API" can only be accessed via Swift +entry points: + +- More potential performance for the shared API. +- Increases binary size and linking time. +- Overriding the method in Objective-C requires teaching Clang to emit + a Swift vtable for the subclass. +- Same default behavior problem as above -- it becomes a decision. + +### Class Model + +All Swift classes are layout-compatible with Objective-C classes: + +- Necessary for `id`-compatibility. +- Increases binary size. + +Only Swift classes marked as "ObjC" (or descending from an Objective-C +class) are layout-compatible with Objective-C classes; other classes are +not: + +- Requires framework authors to make an arbitrary decision. +- Changing the API mode *may* break binary compatibility (consider a + Swift subclass that is not generating Objective-C + class information). + +### Subclassing Model + +*Requirement: can subclass Objective-C objects from Swift.* + +All Swift classes can be subclassed from Objective-C: + +- Potentially increases binary size. +- Requires teaching Clang to emit Swift vtables. + +Only Swift classes marked as "ObjC" (or descending from an Objective-C +class) are subclassable in Objective-C: + +- Probably *still* requires teaching Clang to emit Swift vtables. +- Requires framework authors to make an arbitrary decision that may + not be ABI-future-proof. + +### Method Overriding Model + +*Requirement: Swift classes can override any Objective-C methods.* + +Methods marked as "overrideable API" only have Objective-C entry points: + +- Less to think about, maximum compatibility. +- Penalizes future Swift clients (and potentially + Objective-C clients?). + +Methods marked as "overridable API" have both Objective-C and Swift +entry points: + +- Requires teaching Clang to emit Swift vtables. +- Increases binary size and link time. + +Methods marked as "overrideable API" have only Swift entry points: + +- Requires teaching Clang to emit Swift vtables. +- Later exposing this method to Objective-C in a subclass may be + awkward? + +Attributes for Objective-C Support +---------------------------------- + +`@objc` + +: - When applied to classes, directs the compiler to emit + Objective-C metadata for this class. Additionally, if no + superclass is specified, the superclass is implicitly `NSObject` + rather than the default `swift.Object`. Note that Objective-C + class names must be unique across the entire program, not just + within a single namespace or module. [^5] + - When applied to methods, directs the compiler to emit an + Objective-C entry point and entry in the Objective-C method list + for this method. + - When applied to properties, directs the compiler to emit + Objective-C methods `-`*foo* and `-set`*Foo*`:`, which wrap the + getter and setter for the property. + - When applied to protocols, directs the compiler to emit + Objective-C metadata for this protocol. Objective-C protocols + may contain optional methods. Method definitions for an + Objective-C protocol conformance are themselves implicitly + `@objc`. + + This attribute is inherited (in all contexts). + +`@nonobjc` + +: - When applied to methods, properties, subscripts or constructors, + override the implicit inheritance of `@objc`. + - Only valid if the declaration was implicitly `@objc` as a result + of the class or one of the class's superclasses being `@obj` -- + not permitted on protocol conformances. + - It is permitted to override a `@nonobjc` method with a method + marked as `@objc`; overriding an `@objc` (or implicitly `@objc`) + method with a `@nonobjc` method is not allowed. + - It is an error to combine `@nonobjc` with `dynamic`, `@IBOutlet` + or `@NSManaged`. + + This attribute is inherited. + +`@IBOutlet` + +: Can only be applied to properties. This marks the property as being + exposed as an outlet in Interface Builder. **In most cases,** + [outlets should be weak + properties](http://developer.apple.com/library/mac/documentation/Cocoa/Conceptual/LoadingResources/CocoaNibs/CocoaNibs.html#//apple_ref/doc/uid/10000051i-CH4-SW6). + + *The simplest implementation of this is to have* `@IBOutlet` *cause + an* *Objective-C getter and setter to be emitted, but this is* not + *part of* `@IBOutlet`'s *contract.* + + This attribute is inherited. + +`@IBAction` + +: Can only be applied to methods, which must have a signature matching + the requirements for target/action methods on the current platform. + This marks the method as being a potential action in + Interface Builder. + + *The simplest implementation of this is to have* `@IBAction` *imply* + `@objc`, *and this may be the* only *viable implementation given how + the* *responder chain works. For example, a window's delegate is + part of the* *responder chain, even though it does not subclass* + `NSResponder` *and may* *not be an Objective-C class at all. Still, + this is* not *part of* `@IBAction`'s *contract.* + + This attribute is inherited. + +Level 1: Message-passing +------------------------ + +*Assuming an object is known to be a Swift object or an Objective-C +object at compile-time, what does it take to send a message from one to +the other?* + +### ARC + +> By default, objects are passed to and returned from Objective-C +> methods as +0 (i.e. non-owned objects). The caller does not have to do +> anything to release returned objects, though if they wish to retain +> them they may be able to steal them out of the top autorelease pool. +> (In practice, the caller *does* retain the arguments for the duration +> of the method anyway, unless it can be proven that nothing interferes +> with the lifetime of the object between the load and the call.) +> +> Objective-C methods from certain method families do return +1 objects, +> as do methods explicitly annotated with the `ns_returns_retained` +> attribute. +> +> All Swift class objects (i.e. as opposed to structs) are returned as +> +1 (i.e. owned objects). The caller is responsible for releasing them. + +Swift methods that are exposed as Objective-C methods will have a +wrapper function (thunk) that is responsible for retaining all (object) +arguments and autoreleasing the return value. + +*Swift methods willnot*\* be exposed as\* `ns_returns_retained` because +they should behave like Objective-C methods when called through an\* +`id`. + +### Arguments + +> Objective-C currently requires that the first argument be `self` and +> the second be `_cmd`. The explicit arguments to a method come after +> `_cmd`. +> +> Swift only requires that the first argument be `self`. The explicit +> arguments come after `self`. + +The thunk mentioned above can shift all arguments over...which doesn't +really cost anything extra since we already have to retain all the +arguments. + +### Output Parameters + +> Because Objective-C does not have tuples, returning multiple values is +> accomplished through the use of pointer-to-object-pointer parameters, +> such as `NSError **`. Additionally, objects returned through these +> parameters are conventionally autoreleased, though ARC allows this to +> be specified explicitly. +> +> Swift has tuples and does not have pointers, so the natural way to +> return multiple values is to return a tuple. The retain-count issue is +> different here: with ARC, the tuple owns the objects in it, and the +> caller owns the tuple. +> +> Swift currently also has `[inout]` arguments. Whether or not these +> will be exposed to users and/or used for Objective-C out parameters is +> still undecided. + +*This issue has not been resolved, but it only affects certain API.* + +### Messaging `nil` + +> In Objective-C, the result of messaging `nil` is defined to be a +> zero-filled value of the return type. For methods that return an +> object, the return value is also `nil`. Methods that return non-POD +> C++ objects attempt to default-construct the object if the receiver is +> `nil`. +> +> In Swift, messaging `nil` is undefined, and hoped to be defined away +> by the type system through liberal use of some `Optional` type. +> +> - I've seen other languages explicitly request the Objective-C +> behavior using `foo.?bar()`, though that's not the prettiest +> syntax in the world. -Jordan + +As long as the implementation of `Optional` is layout-compatible with an +object pointer, and an absent `Optional` is represented with a null +pointer, this will Just Work™. + +### Overloading + +> In Objective-C, methods cannot be overloaded. +> +> In Swift, methods can have the exact same name but take arguments of +> different types. +> +> Note that in Swift, all parameters after the first are part of the +> method name, unless using the "selector syntax" for defining methods: +> +> // 1. foo:baz: +> func foo(Int bar, Int baz); +> +> // 2. foo:qux: +> func foo(Int bar, Int qux); +> +> // 3. foo:qux: (same as above) +> func foo(Int bar) qux(Int quux); +> +> // 4. foo:baz: (but different type!) +> func foo(Int bar, UnicodeScalar baz); +> +> a.foo(1, 2) // ambiguous in Swift (#1 or #2?) +> a.foo(1, baz=2) // calls #1 +> a.foo(1, qux=2) // calls #2/3 (the same method) +> a.foo(1, 'C') // calls #4, not ambiguous in Swift! +> +> [a foo:1 baz:2]; // ambiguous in Objective-C (#1 or #4?) +> [a foo:1 qux:2]; // calls #2/3 (the same method) + +The Swift compiler should not let both \#1 and \#4 be exported to +Objective-C. It should already warn about the ambiguity between \#1 and +\#2 without using named parameters. + +Level 2: Messaging `id` +----------------------- + +*If a Swift object can be referenced with* `id`, *how do you send +messages to* *it?* + +Note: the answer might be "Swift objects can't generally be referenced +with `id`". + +### `isa` Pointers + +> The first word of every Objective-C object is a pointer to its class. +> +> We might want to use a more compact representation for Swift +> objects... + +...but we can't; see below. + +### Method Lookup + +> Objective-C performs method lookup by searching a sequence of maps for +> a given key, called a *selector*. Selectors are pointer-sized and +> uniqued across an entire process, so dynamically-loaded methods with +> the same name as an existing method will have an identical selector. +> Each map in the sequence refers to the set of methods added by a +> category (or the original class). If the lookup fails, the search is +> repeated for the superclass. +> +> Swift performs method lookup by vtable. In order to make these vtables +> non-fragile, the offset into a vtable for a given message is stored as +> a global variable. Rather than chaining searches through different +> message lists to account for inheritance and categories, the container +> for each method is known at compile-time. So the final lookup for a +> given method looks something like this: +> +> vtable[SUBCLASS_OFFSET + METHOD_OFFSET] + +Swift class objects will have `isa` pointers, and those `isa` pointers +will have an Objective-C method list at the very least, and probably a +method cache as well. The methods in this list will refer to the +Objective-C-compatible wrappers around Swift methods described above. + +The other words in the `isa` structure may not be used in the same way +as they are in Objective-C; only `objc_msgSend` has to avoid +special-casing Swift objects. Most of the other runtime functions can +probably do a check to see if they are dealing with a Swift class, and +if so fail nicely. + +Level 3a: Adopting Objective-C Protocols in Swift +------------------------------------------------- + +- Bare minimum for implementing an AppKit/UIKit app in Swift. +- Essentially the same as emitting any other Objective-C methods, plus + making `-conformsToProtocol:` and `+conformsToProtocol:` + work properly. + +Level 3b: Adopting Swift Protocols in Objective-C +------------------------------------------------- + +- Requires generating both Swift and Objective-C entry points + from Clang. +- Requires generating Swift protocol vtables. + +*Note: including protocol implementations is essentially the same as +implicitly adding an extension (section 5a).* + +Level 4a: Subclassing Objective-C Classes in Swift +-------------------------------------------------- + +*To be written.* + +- Basically necessary for implementing an AppKit/UIKit app in Swift. +- Requires generating Objective-C-compatible method lists. +- When a new method is marked as API, does it automatically get the + Objective-C calling conventions by default? (See + "Tradeoffs" section.) + +Level 4b: Subclassing Swift Classes in Objective-C +-------------------------------------------------- + +*To be written.* + +- May require generating Swift vtables. + + Alternative: if a method is exposed for overriding, it only gets an + Objective-C entry point. (Downsides: performance, other platforms + will hate us.) + + Alternative: only Swift classes with an Objective-C class in their + hierarchy can be subclassed in Objective-C. Any overridden methods + must be exposed as Objective-C already. (Downsides: framework + authors could forget to inherit from NSObject, Swift code is + penalized ahead of time.) + + Alternative: only Swift classes with an Objective-C class in their + hierarchy are *visible* in Objective-C. All other Swift objects + are opaque. (Downsides: same as above.) + +Level 5a: Adding Extensions to Objective-C Classes in Swift +----------------------------------------------------------- + +*To be written.* + +- May require generating Objective-C-compatible method lists. +- Less clear what the *default* calling convention should be for + new methods. + +Level 5b: Adding Categories to Swift Classes in Objective-C +----------------------------------------------------------- + +*To be written.* + +- Does not actually *require* generating Swift vtables. But we could + if we wanted to expose Swift entry points for these methods as well. +- Does require an Objective-C-compatible `isa` to attach the new + method list to. + +Level 6: Dynamic Subclassing +---------------------------- + +*To be written, but probably not an issue...it's mostly the same as +statically subclassing, right?* + +Level 7: Method Swizzling +------------------------- + +I'm okay with just saying "no" to this one. + +[^1]: Really, "All Swift objects on OS X and iOS". Presumably a Swift + compiler on another system wouldn't bother to emit the Objective-C + isa info. + +[^2]: Dave is working out an object and class layout scheme that will + minimize the performance cost of emitting both the Objective-C isa + and a Swift vtable. It is entirely possible that from the Swift + perspective, the Objective-C isa is just an opaque "vtable slice" + that is fixed at offset 0. + +[^3]: ...or any other Objective-C class, including alternate roots like + NSProxy. Most likely this will be implemented with an inherited + attribute `[objc]` on the class, which would even allow Swift to + create Objective-C root classes. + +[^4]: If you explicitly want to expose a Swift method to Objective-C, + but it is not part of an existing protocol, you can mark the method + as "API" and include the `[objc]` attribute: + + // Note: This syntax is not final! + func [API, objc] accessibilityDescription { + return "\(self.givenName) \(self.familyName)" + } + +[^5]: I'm not really sure what to do about uniquing Objective-C class + names. Maybe eventually \[objc\] will take an optional argument + specifying the Objective-C-equivalent name. diff --git a/docs/archive/Objective-C Interoperability.rst b/docs/archive/Objective-C Interoperability.rst deleted file mode 100644 index 5524eb4cfa71b..0000000000000 --- a/docs/archive/Objective-C Interoperability.rst +++ /dev/null @@ -1,653 +0,0 @@ -:orphan: - -.. @raise litre.TestsAreMissing - -============================ -Objective-C Interoperability -============================ - -This document tracks the differences between the Swift and Objective-C ABIs and -class models, and what it would take to merge the two as much as possible. The -format of each section lays out the differences between Swift and Objective-C, -then describes what needs to happen for a user to mix the two seamlessly. - -.. warning:: This document was used in planning Swift 1.0; it has not been kept - up to date and does not describe the current or planned behavior of Swift. - - -.. contents:: - -Terminology used in this document: - -- ``id``-compatible: something that can be assigned to an ``id`` variable and - sent messages using ``objc_msgSend``. In practice, this probably means - implementing the ``NSObject`` protocol, since most of Cocoa doesn't check - whether something implements ``NSObject`` before sending a message like - ``-class``. - -- Objective-C isa: something that identifies the class of an Objective-C object, - used by ``objc_msgSend``. To say a Swift object has an Objective-C isa does - *not* mean that a fully-formed Objective-C runtime class structure is - generated for the Swift class; it just means that (1) the header of the Swift - object "looks like" an Objective-C object, and (2) the parts of an Objective-C - class used by the ``objc_msgSend`` "fast path" are the same. - - -Design -====== - -All Swift objects [#]_ will be ``id``-compatible and will have an Objective-C -isa, on the assumption that you want to be able to put them in an array, set -them as represented objects, etc. [#]_ - -Swift classes that inherit from NSObject (directly or indirectly [#]_) behave -exactly like Objective-C classes from the perspective of Objective-C source. -All methods marked as "API" in Swift will have dual entry points exposed by -default. Methods not marked as "API" will not be exposed to Objective-C at all. -Instances of these classes can be used like any other Objective-C objects. - -Subclassing a "Swift NSObject class" in Objective-C requires a bit of extra -work: generating Swift vtables. We haven't decided how to do this: - -- Clang could be taught about Swift class layout. -- The Clang driver could call out to the Swift compiler to do this. Somehow. -- The runtime could fill in the vtable from the Objective-C isa list at class - load time. (This could be necessary anyway to support dynamic subclassing... - which we may or may not do.) - -Swift classes that do not inherit from NSObject are not visible from -Objective-C. Their instances can be manipulated as ``id``, or via whatever -protocols they may implement. - -:: - - class AppController : NSApplicationDelegate { - func applicationDidFinishLaunching(notification : NSNotification) { - // do stuff - } - } - - // Use 'id ' in Objective-C. - -Like "Swift NSObject classes", though, "pure" Swift classes will still have an -isa, and any methods declared in an Objective-C protocol will be emitted with -dual entry points. - - -.. [#] Really, "All Swift objects on OS X and iOS". Presumably a Swift compiler - on another system wouldn't bother to emit the Objective-C isa info. -.. [#] Dave is working out an object and class layout scheme that will minimize - the performance cost of emitting both the Objective-C isa and a Swift vtable. - It is entirely possible that from the Swift perspective, the Objective-C isa - is just an opaque "vtable slice" that is fixed at offset 0. -.. [#] ...or any other Objective-C class, including alternate roots like - NSProxy. Most likely this will be implemented with an inherited attribute - ``[objc]`` on the class, which would even allow Swift to create Objective-C - root classes. - - -Use Cases -========= - -*Unfinished and undetailed.* - -Simple Application Writer -------------------------- - -I want to write my new iOS application in Swift, using all the Objective-C -frameworks that come with iOS. - -Guidelines: - -Everything should Just Work™. There should be no need to subclass NSObject -anywhere in your program, unless you are specifically specializing a class in -the Cocoa Touch frameworks. - - -Intermediate Application Writer -------------------------------- - -I want to write my new application in Objective-C, but there's a really nice -Swift framework I want to use. - -Guidelines: - -- Not all Swift methods in the framework may be available in Objective-C. You - can work around this by adding *extensions* to the Swift framework classes to - expose a more Objective-C-friendly interface. You will need to mark these new - methods as "API" in order to make them visible to Objective-C. -- "Pure" Swift classes will not be visible to Objective-C at all. You will have - to write a wrapper class (or wrapper functions) in Swift if you want to use - the features of these classes directly. However, you can still treat them - like any other objects in your program (store them in ``id`` variables, - Objective-C collections, etc). - - -Transitioning Application Writer --------------------------------- - -I have an existing Objective-C application, and I want to convert it -piece-by-piece to Swift. - -Guidelines: - -- Swift is different from Objective-C in that methods in Swift classes are not - automatically usable from everywhere. If your Swift class inherits from - NSObject, marking your methods as "API" will allow them to be called from - Objective-C code. A Swift class that does not inherit from NSObject will only - respond to messages included in its adopted protocols. [#]_ -- Once you have finished transitioning to Swift, go through your classes and - remove the "API" marker from any methods that do not need to be accessed from - Objective-C. Remove NSObject as a superclass from any classes that do not need - to be accessed from Objective-C. Both of these allow the compiler to be more - aggressive in optimizing your program, potentially making it both smaller and - faster. - -.. [#] If you explicitly want to expose a Swift method to Objective-C, but it - is not part of an existing protocol, you can mark the method as "API" and - include the ``[objc]`` attribute:: - - // Note: This syntax is not final! - func [API, objc] accessibilityDescription { - return "\(self.givenName) \(self.familyName)" - } - -New Framework Writer --------------------- - -I want to write a framework that can be used by anyone. - -Requirements: - -- Can call (at least some) Swift methods from Objective-C. - - -Intermediate Framework Writer ------------------------------ - -I have an existing Objective-C framework that I want to move to Swift. - -Requirements: - -- Can subclass Objective-C classes in Swift. -- Can call (at least some) Swift methods from Objective-C. - -Decisions: - -- Should I expose Swift entry points as API? -- If so, should they be essentially the same as the Objective-C entry points, or - should I have a very different interface that's more suited for Swift (and - easily could be "better")? - - -End User --------- - -- Things should be fast. -- Things should not take a ton of memory. - - -Nice to Have (uncategorized) ----------------------------- - -- Can write a Swift extension for an Objective-C class. -- Can write a Swift extension for an Objective-C class that adopts an - Objective-C protocol. -- Can write a Swift extension for an Objective-C class that exposes arbitrary - new methods in Objective-C. - - -Tradeoffs -========= - -This section discusses models for various runtime data structures, and the -tradeoffs for making Swift's models different from Objective-C. - -Messaging Model ---------------- - -Everything is ``id``-compatible: - -- Less to think about, maximum compatibility. -- Every Swift object must have an Objective-C isa. - -Non-NSObjects are messageable but not ``id``-compatible: - -- Cannot assign Swift objects to ``id`` variables. -- Cannot put arbitrary Swift objects in NSArrays. -- Potentially confusing: "I can message it but I can't put it in an ``id``??" -- Clang must be taught how to message Swift objects and manage their retain - counts. -- On the plus side, then non-NSObjects can use Swift calling conventions. -- Requires framework authors to make an arbitrary decision that may not be - ABI-future-proof. - -Non-NSObjects are opaque: - -- Can be passed around, but not manipulated. -- ...but Clang probably *still* has to be taught how to manage the retain count - of an opaque Swift object, and doing so in the same way as dispatch_queue_t - and friends may be dangerous (see -- it's pretending they're - NSObjects, which they are) -- Requires framework authors to make an arbitrary decision that may not be - ABI-future-proof. - - -Method Model ------------- - -*This only affects methods marked as "API" in some way. Assume for now that all -methods use types shared by both Objective-C and Swift, and that calls within -the module can still be optimized away. Therefore, this discussion only applies -to frameworks, and specifically the use of Swift methods from outside of the -module in which they are defined.* - -Every method marked as API can *only* be accessed via Objective-C entry points: - -- Less to think about, maximum compatibility. -- Penalizes future Swift clients (and potentially Objective-C clients?). - -Every method marked as API can be accessed both from Objective-C and Swift: - -- Maximum potential performance. -- Increases binary size and linking time. -- If this is a framework converted to Swift, clients that link against the - Swift entry points are no longer backwards-compatible. And it's hard to know - what you did wrong here. -- Overriding the method in Objective-C requires teaching Clang to emit a Swift - vtable for the subclass. - -Methods marked as "ObjC API" can only be accessed via Objective-C entry points; -methods marked as "Swift API" can only be accessed via Swift entry points: - -- Changing the API mode breaks binary compatibility. -- Obviously this attribute is inherited -- overriding an Objective-C method - should produce a new Objective-C entry point. What is the default for new - methods, though? Always Swift? Always Objective-C? Based on the class model - (see below)? Specified manually? - -Methods marked as "ObjC API" can be accessed both from Objective-C and Swift; -methods marked as "Swift API" can only be accessed via Swift entry points: - -- More potential performance for the shared API. -- Increases binary size and linking time. -- Overriding the method in Objective-C requires teaching Clang to emit a Swift - vtable for the subclass. -- Same default behavior problem as above -- it becomes a decision. - - -Class Model ------------ - -All Swift classes are layout-compatible with Objective-C classes: - -- Necessary for ``id``-compatibility. -- Increases binary size. - -Only Swift classes marked as "ObjC" (or descending from an Objective-C class) -are layout-compatible with Objective-C classes; other classes are not: - -- Requires framework authors to make an arbitrary decision. -- Changing the API mode *may* break binary compatibility (consider a Swift - subclass that is not generating Objective-C class information). - - -Subclassing Model ------------------ - -*Requirement: can subclass Objective-C objects from Swift.* - -All Swift classes can be subclassed from Objective-C: - -- Potentially increases binary size. -- Requires teaching Clang to emit Swift vtables. - -Only Swift classes marked as "ObjC" (or descending from an Objective-C class) -are subclassable in Objective-C: - -- Probably *still* requires teaching Clang to emit Swift vtables. -- Requires framework authors to make an arbitrary decision that may not be - ABI-future-proof. - - -Method Overriding Model ------------------------ - -*Requirement: Swift classes can override any Objective-C methods.* - -Methods marked as "overrideable API" only have Objective-C entry points: - -- Less to think about, maximum compatibility. -- Penalizes future Swift clients (and potentially Objective-C clients?). - -Methods marked as "overridable API" have both Objective-C and Swift entry -points: - -- Requires teaching Clang to emit Swift vtables. -- Increases binary size and link time. - -Methods marked as "overrideable API" have only Swift entry points: - -- Requires teaching Clang to emit Swift vtables. -- Later exposing this method to Objective-C in a subclass may be awkward? - - -Attributes for Objective-C Support -================================== - -``@objc`` - - When applied to classes, directs the compiler to emit Objective-C metadata - for this class. Additionally, if no superclass is specified, the superclass - is implicitly ``NSObject`` rather than the default ``swift.Object``. - Note that Objective-C class names must be unique across the entire program, - not just within a single namespace or module. [#]_ - - When applied to methods, directs the compiler to emit an Objective-C entry - point and entry in the Objective-C method list for this method. - - When applied to properties, directs the compiler to emit Objective-C methods - ``-``\ *foo* and ``-set``\ *Foo*\ ``:``, which wrap the getter and setter - for the property. - - When applied to protocols, directs the compiler to emit Objective-C metadata - for this protocol. Objective-C protocols may contain optional methods. - Method definitions for an Objective-C protocol conformance are themselves - implicitly ``@objc``. - - This attribute is inherited (in all contexts). - -``@nonobjc`` - - When applied to methods, properties, subscripts or constructors, override the - implicit inheritance of ``@objc``. - - Only valid if the declaration was implicitly ``@objc`` as a result of the - class or one of the class's superclasses being ``@obj`` -- not permitted on - protocol conformances. - - It is permitted to override a ``@nonobjc`` method with a method marked as - ``@objc``; overriding an ``@objc`` (or implicitly ``@objc``) method with a - ``@nonobjc`` method is not allowed. - - It is an error to combine ``@nonobjc`` with ``dynamic``, ``@IBOutlet`` or - ``@NSManaged``. - - This attribute is inherited. - -``@IBOutlet`` - Can only be applied to properties. This marks the property as being exposed - as an outlet in Interface Builder. **In most cases,** - `outlets should be weak properties`__. - - *The simplest implementation of this is to have* ``@IBOutlet`` *cause an* - *Objective-C getter and setter to be emitted, but this is* not *part of* - ``@IBOutlet``'s *contract.* - - This attribute is inherited. - -``@IBAction`` - Can only be applied to methods, which must have a signature matching the - requirements for target/action methods on the current platform. - This marks the method as being a potential action in Interface Builder. - - *The simplest implementation of this is to have* ``@IBAction`` *imply* - ``@objc``, *and this may be the* only *viable implementation given how the* - *responder chain works. For example, a window's delegate is part of the* - *responder chain, even though it does not subclass* ``NSResponder`` *and may* - *not be an Objective-C class at all. Still, this is* not *part of* - ``@IBAction``'s *contract.* - - This attribute is inherited. - -.. [#] I'm not really sure what to do about uniquing Objective-C class names. - Maybe eventually [objc] will take an optional argument specifying the - Objective-C-equivalent name. - -__ http://developer.apple.com/library/mac/documentation/Cocoa/Conceptual/LoadingResources/CocoaNibs/CocoaNibs.html#//apple_ref/doc/uid/10000051i-CH4-SW6 - - -Level 1: Message-passing -======================== - -*Assuming an object is known to be a Swift object or an Objective-C object at -compile-time, what does it take to send a message from one to the other?* - - -ARC ---- - - By default, objects are passed to and returned from Objective-C methods as +0 - (i.e. non-owned objects). The caller does not have to do anything to release - returned objects, though if they wish to retain them they may be able to steal - them out of the top autorelease pool. (In practice, the caller *does* retain - the arguments for the duration of the method anyway, unless it can be proven - that nothing interferes with the lifetime of the object between the load and - the call.) - - Objective-C methods from certain method families do return +1 objects, as do - methods explicitly annotated with the ``ns_returns_retained`` attribute. - - All Swift class objects (i.e. as opposed to structs) are returned as +1 (i.e. - owned objects). The caller is responsible for releasing them. - -Swift methods that are exposed as Objective-C methods will have a wrapper -function (thunk) that is responsible for retaining all (object) arguments and -autoreleasing the return value. - -*Swift methods will **not** be exposed as* ``ns_returns_retained`` because they -should behave like Objective-C methods when called through an* ``id``. - - -Arguments ---------- - - Objective-C currently requires that the first argument be ``self`` and the - second be ``_cmd``. The explicit arguments to a method come after ``_cmd``. - - Swift only requires that the first argument be ``self``. The explicit - arguments come after ``self``. - -The thunk mentioned above can shift all arguments over...which doesn't really -cost anything extra since we already have to retain all the arguments. - - -Output Parameters ------------------ - - Because Objective-C does not have tuples, returning multiple values is - accomplished through the use of pointer-to-object-pointer parameters, such as - ``NSError **``. Additionally, objects returned through these parameters are - conventionally autoreleased, though ARC allows this to be specified - explicitly. - - Swift has tuples and does not have pointers, so the natural way to return - multiple values is to return a tuple. The retain-count issue is different - here: with ARC, the tuple owns the objects in it, and the caller owns the - tuple. - - Swift currently also has ``[inout]`` arguments. Whether or not these will be - exposed to users and/or used for Objective-C out parameters is still - undecided. - -*This issue has not been resolved, but it only affects certain API.* - - -Messaging ``nil`` ------------------ - - In Objective-C, the result of messaging ``nil`` is defined to be a zero-filled - value of the return type. For methods that return an object, the return value - is also ``nil``. Methods that return non-POD C++ objects attempt to - default-construct the object if the receiver is ``nil``. - - In Swift, messaging ``nil`` is undefined, and hoped to be defined away by the - type system through liberal use of some ``Optional`` type. - - - I've seen other languages explicitly request the Objective-C behavior using - ``foo.?bar()``, though that's not the prettiest syntax in the world. - -Jordan - -As long as the implementation of ``Optional`` is layout-compatible with an -object pointer, and an absent ``Optional`` is represented with a null pointer, -this will Just Work™. - - -Overloading ------------ - In Objective-C, methods cannot be overloaded. - - In Swift, methods can have the exact same name but take arguments of different - types. - - Note that in Swift, all parameters after the first are part of the method - name, unless using the "selector syntax" for defining methods:: - - // 1. foo:baz: - func foo(Int bar, Int baz); - - // 2. foo:qux: - func foo(Int bar, Int qux); - - // 3. foo:qux: (same as above) - func foo(Int bar) qux(Int quux); - - // 4. foo:baz: (but different type!) - func foo(Int bar, UnicodeScalar baz); - - a.foo(1, 2) // ambiguous in Swift (#1 or #2?) - a.foo(1, baz=2) // calls #1 - a.foo(1, qux=2) // calls #2/3 (the same method) - a.foo(1, 'C') // calls #4, not ambiguous in Swift! - - [a foo:1 baz:2]; // ambiguous in Objective-C (#1 or #4?) - [a foo:1 qux:2]; // calls #2/3 (the same method) - -The Swift compiler should not let both #1 and #4 be exported to Objective-C. -It should already warn about the ambiguity between #1 and #2 without using -named parameters. - - -Level 2: Messaging ``id`` -========================= - -*If a Swift object can be referenced with* ``id``, *how do you send messages to* -*it?* - -Note: the answer might be "Swift objects can't generally be referenced with -``id``". - - -``isa`` Pointers ----------------- - The first word of every Objective-C object is a pointer to its class. - - We might want to use a more compact representation for Swift objects... - -...but we can't; see below. - - -Method Lookup -------------- - Objective-C performs method lookup by searching a sequence of maps for a - given key, called a *selector*. Selectors are pointer-sized and uniqued - across an entire process, so dynamically-loaded methods with the same name as - an existing method will have an identical selector. Each map in the sequence - refers to the set of methods added by a category (or the original class). If - the lookup fails, the search is repeated for the superclass. - - Swift performs method lookup by vtable. In order to make these vtables - non-fragile, the offset into a vtable for a given message is stored as a - global variable. Rather than chaining searches through different message - lists to account for inheritance and categories, the container for each - method is known at compile-time. So the final lookup for a given method looks - something like this:: - - vtable[SUBCLASS_OFFSET + METHOD_OFFSET] - -Swift class objects will have ``isa`` pointers, and those ``isa`` pointers will -have an Objective-C method list at the very least, and probably a method cache -as well. The methods in this list will refer to the Objective-C-compatible -wrappers around Swift methods described above. - -The other words in the ``isa`` structure may not be used in the same way as they -are in Objective-C; only ``objc_msgSend`` has to avoid special-casing Swift -objects. Most of the other runtime functions can probably do a check to see if -they are dealing with a Swift class, and if so fail nicely. - - -Level 3a: Adopting Objective-C Protocols in Swift -================================================= - -- Bare minimum for implementing an AppKit/UIKit app in Swift. -- Essentially the same as emitting any other Objective-C methods, plus making - ``-conformsToProtocol:`` and ``+conformsToProtocol:`` work properly. - - -Level 3b: Adopting Swift Protocols in Objective-C -================================================= - -- Requires generating both Swift and Objective-C entry points from Clang. -- Requires generating Swift protocol vtables. - -*Note: including protocol implementations is essentially the same as implicitly -adding an extension (section 5a).* - - -Level 4a: Subclassing Objective-C Classes in Swift -================================================== - -*To be written.* - -- Basically necessary for implementing an AppKit/UIKit app in Swift. -- Requires generating Objective-C-compatible method lists. -- When a new method is marked as API, does it automatically get the Objective-C - calling conventions by default? (See "Tradeoffs" section.) - - -Level 4b: Subclassing Swift Classes in Objective-C -================================================== - -*To be written.* - -- May require generating Swift vtables. - - Alternative: if a method is exposed for overriding, it only gets an - Objective-C entry point. (Downsides: performance, other platforms will hate - us.) - - Alternative: only Swift classes with an Objective-C class in their hierarchy - can be subclassed in Objective-C. Any overridden methods must be exposed as - Objective-C already. (Downsides: framework authors could forget to inherit - from NSObject, Swift code is penalized ahead of time.) - - Alternative: only Swift classes with an Objective-C class in their hierarchy - are *visible* in Objective-C. All other Swift objects are opaque. - (Downsides: same as above.) - - -Level 5a: Adding Extensions to Objective-C Classes in Swift -=========================================================== - -*To be written.* - -- May require generating Objective-C-compatible method lists. -- Less clear what the *default* calling convention should be for new methods. - - -Level 5b: Adding Categories to Swift Classes in Objective-C -=========================================================== - -*To be written.* - -- Does not actually *require* generating Swift vtables. But we could if we - wanted to expose Swift entry points for these methods as well. - -- Does require an Objective-C-compatible ``isa`` to attach the new method list - to. - - -Level 6: Dynamic Subclassing -============================ - -*To be written, but probably not an issue...it's mostly the same as statically -subclassing, right?* - - -Level 7: Method Swizzling -========================= - -I'm okay with just saying "no" to this one. - diff --git a/docs/archive/Resilience.md b/docs/archive/Resilience.md new file mode 100644 index 0000000000000..9ca45897fc09e --- /dev/null +++ b/docs/archive/Resilience.md @@ -0,0 +1,616 @@ +Resilience +========== + +> **warning** +> +> This is a very early design document discussing the feature of + +> Resilience. It should not be taken as a plan of record. + +Introduction +------------ + +One of Swift's primary design goals is to allow efficient execution of +code without sacrificing load-time abstraction of implementation. + +Abstraction of implementation means that code correctly written against +a published interface will correctly function when the underlying +implementation changes to anything which still satisfies the original +interface. There are many potential reasons to provide this sort of +abstraction. Apple's primary interest is in making it easy and painless +for our internal and external developers to improve the ecosystem of +Apple products by creating good and secure programs and libraries; +subtle deployment problems and/or unnecessary dependencies on the +behavior of our implementations would work against these goals. + +Almost all languages provide some amount of abstraction of +implementation. For example, functions are usually opaque data types +which fully abstract away the exact sequence of operations performed. +Similarly, adding a new field to a C struct does not break programs +which refer to a different field — those programs may need to be +recompiled, but once recompiled, they should continue to work. (This +would not necessarily be true if, say, fields were accessed by index +rather than by name.) + +Components +---------- + +Programs and libraries are not distributed as a legion of source files +assembled independently by the end user. Instead, they are packaged into +larger components which are distributed and loaded as a unit. Each +component that comprises a program may depend on some number of other +components; this graph of dependencies can be assumed to be acyclic. + +Because a component is distributed as a unit, ABI resilience within the +component is not required. It may still help to serve as a build- time +optimization, but Swift aims to offer substantially better build times +than C/C++ programs due to other properties of the language (the module +system, the lack of a preprocessor, the instantiation model, etc.). + +Components may be defined as broadly as an entire operating system or as +narrowly as the build products of an individual team. The development +process of a very large product (like an operating system) may +discourage a model which requires almost the entire system to be +recompiled when a low-level component is changed, even if recompilation +is relatively fast. Swift aims to reduce the need for unnecessary +abstraction penalties where possible, but to allow essentially any model +that conforms to the basic rule of acyclicity. + +Abstraction Throughout the Program Lifecycle +-------------------------------------------- + +Most languages provide different amounts of abstraction at different +stages in the lifecycle of a program. The stages are: + +1. Compilation, when individual source files are translated into a form + suitable for later phases. The abstractions of C/C++ object layout + and C++ virtual table layout are only provided until this stage; + adding a field to a struct or a virtual method to a class + potentially require all users of those interfaces to be recompiled. +2. Bundling, when compiled source files are combined into a larger + component; this may occur in several stages, e.g. when object files + are linked into a shared library which is then bundled up into an + OS distribution. Objective-C classes in the non-fragile ABI are laid + out by special processing that occurs during bundling; this is done + just as an optimization and the runtime can force additional layout + changes at execution time, but if that were not true, the + abstraction would be lost at this stage. +3. Installation, when bundled components are placed on the system where + they will be executed. This is generally a good stage in which to + lose abstraction if implementations are always installed + before interfaces. This is the first stage whose performance is + visible to the end user, but unless installation times are absurdly + long, there is little harm to doing extra processing here. +4. Loading, when a component is loaded into memory in preparation for + executing it. Objective-C performs class layout at this stage; after + loading completes, no further changes to class layout are possible. + Java also performs layout and additional validation of class files + during program loading. Processing performed in this stage delays + the start-up of a program (or plugin or other component). +5. Execution, when a piece of code actually evaluates. Objective-C + guarantees method lookup until this "stage", meaning that it is + always possible to dynamically add methods to a class. Languages + which permit code to be dynamically reloaded must also enforce + abstraction during execution, although JIT techniques can mitigate + some of this expense. Processing performed in this stage directly + slows down the operation. + +Expressivity +------------ + +Certain language capabilities are affected by the choice of when to +break down the abstraction. + +- If the language runtime offers functions to dynamically explore and, + by reflection, use the language structures in a component, some + amount of metadata must survive until execution time. For example, + invoking a function would require the signature and ABI information + about the types it uses to be preserved sufficiently for the + invocation code to reassemble it. +- If the language runtime offers functions to dynamically change or + augment the language structures in a component, the full + abstractions associated with those structures must be preserved + until execution time. For example, if an existing "virtual" method + can be replaced at runtime, no devirtualization is permitted at + compile time and there must be some way to at least map that method + to a vtable offset in all derived classes; and if such a method can + be dynamically added, there must be some ability for method dispatch + to fall back on a dictionary. +- The above is equally true for class extensions in components which + may be loaded or unloaded dynamically. + +Performance +----------- + +The performance costs of abstraction mostly take three forms: + +- the direct cost of many small added indirections +- code-size inflation due to the extra logic to implement these + operations +- diminished potential for more powerful optimizations such as + inlining and code motion + +As mentioned before, we can avoid these costs within components because +classes cannot be extended within components. + +We wish to avoid these costs wherever possible by exploiting the +deployment properties of programs. + +Summary of Design +----------------- + +Our current design in Swift is to provide opt-out load-time abstraction +of implementation for all language features. Alone, this would either +incur unacceptable cost or force widespread opting-out of abstraction. +We intend to mitigate this primarily by designing the language and its +implementation to minimize unnecessary and unintended abstraction: + +- Within the component that defines a language structure, all the + details of its implementation are available. +- When language structures are not exposed outside their defining + components, their implementation is not constrained. +- By default, language structures are not exposed outside their + defining components. This is independently desirable to reduce + accidental API surface area, but happens to also interact well with + the performance design. +- Avoiding unnecessary language guarantees and taking advantage of + that flexibility to limit load-time costs. + +We also intend to provide tools to detect inadvertent changes in +interfaces. + +Components +---------- + +(This is just a sketch and deserves its own design document.) + +Swift will have an integrated build system. This serves several +purposes: + +- it creates a "single source of truth" about the project that can be + shared between tools, +- it speeds up compiles by limiting redundant computation between + files, and +- it gives the compiler information about the boundaries + between components. + +In complex cases, the build process is going to need to be described. +Complex cases include: + +- complex component hierarchies (see below) +- the presence of non-Swift source files (to support directly: .s, .c, + .o, maybe .m, .mm, .cpp) +- a build product other than an executable (to support directly: + executable, .dylib (.framework?), .o, maybe some binary + component distribution) +- library requirements +- deployment requirements +- compilation options more complicated than -On + +This specification file will basically function as the driver interface +to Swift and will probably need a similar host of features, e.g. QA +overrides, inheritance of settings from B&I. Some sort of target-based +programming may also be required. + +Components may be broken down into hierarchies of subcomponents. The +component graph must still be acyclic. + +Every component has a resilience domain , a component (either itself or +an ancestor in its component hierarchy) outside of which resilience is +required. By default, this is the top-level component in its hierarchy. + +Access +------ + +(sketch) + +A lot of code is not intended for use outside the component it appears +in. Here are four levels of access control, along with their proposed +spellings: + +- \[api\] accessible from other components +- \[public\] accessible only from this component (may need finer + grades of control to deal with non-trivial component + hierarchies, e.g. public(somecomponent)) +- \[private\] accessible only from this source file +- \[local\] accessible only from things lexically included in the + containing declaration (may not be useful) + +A language structure's accessibility is inherited by default from its +lexical context. + +The global context (i.e. the default accessibility) is \[public\], i.e. +accessible from this component but not outside it. + +A language structure which is accessible outside the component it +appears in is said to be exported. + +Resilience +---------- + +In general, resilience is the ability to change the implementation of a +language structure without requiring further pre-load-time processing of +code that uses that structure and whose resilience domain does not +include the component defining that structure. + +Resilience does not permit changes to the language structure's +interface. This is a fairly vague standard (that will be spelled out +below), but in general, an interface change is a change which would +cause existing code using that structure to not compile or to compile +using different formal types. + +Language structures may opt out of resilience with an attribute, +\[fragile\]. Deployment versions may be associated with the attribute, +like so: \[fragile(macosx10.7, ios5)\]. It is an interface change to +remove an \[fragile\] attribute, whether versioned or unversioned. It is +an interface change to add an unversioned \[fragile\] attribute. It is +not an interface change to add a versioned \[fragile\] attribute. There +is also a \[resilient\] attribute, exclusive to any form of \[fragile\], +to explicitly describe a declaration as resilient. + +Resilience is lexically inherited. It is not lexically constrained; a +resilient language structure may have fragile sub-structures and vice- +versa. The global context is resilient, although since it is also +\[public\] (and not \[api\]), objects are not in practice constrained by +resilience. + +We intend to provide a tool to automatically detect interface changes. + +Properties of types +------------------- + +A deployment is an abstract platform name and version. + +A type exists on a deployment if: + +- it is a builtin type, or +- it is a function type and its argument and result types exist on the + deployment, or +- it is a tuple type and all of its component types exist on the + deployment, or +- it is a struct, class, or enum type and it does not have an + \[available\] attribute with a later version for a matching + platform name. + +It is an interface change for an exported type to gain an \[available\] +attribute. + +A type is empty if it has a fragile representation (defined below) and: + +- it is a tuple type with no non-empty component types, or +- it is a struct type with no non-empty fields, or +- it is an enum type with one alternative which either carries no data + or carries data with an empty type. + +A type has a fragile representation if: + +- it is a builtin type. The representation should be obvious from + the type. +- it is a function type. The representation is a pair of two pointers: + a valid function pointer, and a potentially null retainable pointer. + See the section on calls for more information. +- it is a tuple type with only fragilely-represented component types. + The representation of a tuple uses the Swift struct + layout algorithm. This is true even if the tuple does not have a + fragile representation. +- it is a class type (that is, a reference struct type). The + representation is a valid retainable pointer. +- it is a fragile struct type with no resilient fields and no fields + whose type is fragilely represented. The representation uses the + Swift struct layout algorithm. + +A type has a universally fragile representation if there is no +deployment of the target platform for which the type exists and is not +fragilely represented. It is a theorem that all components agree on +whether a type has a universal fragile representation and, if so, what +the size, unpadded size, and alignment of that type is. + +Swift's struct layout algorithm takes as input a list of fields, and +does the following: + +1. The fields are ranked: + - The universally fragile fields rank higher than the others. + - If two fields A and B are both universally fragile, + - If no other condition applies, fields that appear earlier in the + original sequence have higher rank. + +2. The size of the structure is initially 0. + + representations and A's type is more aligned than B's type, or + otherwise if A appears before B in the original ordering. + +3. Otherwise. Field A is ranked higher than Field B if + - A has a universal fragile representation and B does not, or + +Swift provides the following types: + +A language structure may be resilient but still define or have a type + +In the following discussion, it will be important to distinguish between +types whose values have a known representation and those which may not. + +Swift provides + +For some structures, it may be important to know that the structure has +never been deployed resiliently, so in general it is considered an +interface change to change a + +Resilience affects pretty much every language feature. + +Execution-time abstraction does not come without cost, and we do not +wish to incur those costs where unnecessary. Many forms of execution- +time abstraction are unnecessary except as a build-time optimization, +because in practice the software is deployed in large chunks that can be +compiled at the same time. Within such a resilience unit , many +execution-time abstractions can be broken down. However, this means that +the resilience of a language structure is context-dependent: it may need +to be accessed in a resilient manner from one resilience unit, but can +be accessed more efficiently from another. A structure which is not +accessible outside its resilience unit is an important exception. A +structure is said to be exported if it is accessible in some theoretical +context outside its resilience unit. + +A structure is said to be resilient if accesses to it rely only on its + +A structure is said to be universally non-resilient if it is non- +resilient in all contexts in which it is accessible. + +Many APIs are willing to selectively "lock down" some of their component +structures, generally because either the form of the structure is +inherent (like a point being a pair of doubles) or important enough to +performance to be worth committing to (such as the accessors of a basic +data structure). This requires an \[unchanging\] annotation and is +equivalent to saying that the structure is universally non-resilient. + +Most language structures in Swift have resilience implications. This +document will need to be updated as language structures evolve and are +enhanced. + +Type categories +--------------- + +For the purposes of this document, there are five categories of type in +Swift. + +**Primitive types**: i1, float32, etc. Nominal types defined by the +implementation. + +**Functions**: () -> int, (NSRect, bool) -> (int, int), etc. +Structural types with + +**Tuples**: (NSRect, bool), (int, int), etc. Structural product types. + +**Named value types**: int, NSRect, etc. Nominal types created by a +variety of language structures. + +**Named reference types**: MyBinaryTree, NSWindow, etc. Nominal types +created by a variety of language structures. + +Primitive types are universally non-resilient. + +Function types are universally non-resilient (but see the section on +calls). + +Tuple types are non-resilient if and only if all their component types +are non-resilient. + +Named types declared within a function are universally non-resilient. + +Named types with the \[unchanging\] annotation are universally +non-resilient. Problem, because of the need/desire to make things depend +on whether a type is universally non-resilient. Makes it impossible to +add \[unchanging\] without breaking ABI. See the call section. + +All other named types are non-resilient only in contexts that are in the +same resilient unit as their declaring file. + +Storage +------- + +Primitive types always have the primitive's size and alignment. + +Named reference types always have the size and alignment of a single +pointer. + +Function types always have the size and alignment of two pointers, the +first being a maximally-nonresilient function pointer (see the section +on calls) and the second being a retain/released pointer. + +If a tuple type is not universally non-resilient, its elements are +stored sequentially using C struct layout rules. Layout must be computed +at runtime. Separate storage is not really a feasible alternative. + +Named types +----------- + +It is an error to place the \[unchanging\] annotation on any of these +types: + +- a struct type with member types that are not universally non- + resilient +- an enum type with an enumerator whose type is not universally non- + resilient +- a class extension +- a class whose primary definition contains a field which is not + universally non-resilient + +Classes +------- + +It is an error to place the \[unchanging\] annotation on a class +extension. + +It is an error to place the \[unchanging\] annotation on a class whose +primary definition contains a field whose type is potentially resilient +in a context where the class is accessible. That is, if the class is +exported, all of its fields must be universally non- resilient. If it is +not exported, all of its fields must be non- resilient within its +resilience unit. + +It is allowed to add fields to an \[unchanging\] class in a class +extension. Such fields are always side-stored, even if they are declared +within the same resilience unit. + +Objects +------- + +Right now, all types in Swift are "first-class", meaning that there is a +broad range of generic operations can be + +1. the size and layout of first-class objects: + - local variables + - global variables + - dynamically\*allocated objects + - member sub\*objects of a structure + - base sub\*objects of a class + - element sub\*objects of an array + - parameters of functions + - results of functions + +2. the set of operations on an object: + - across all protocols + - for a particular protocol (?) + +3. the set of operations on an object + - ... + +ABI resilience means not making assumptions about language entities +which limit the ability of the maintainers of those entities to change +them later. Language entities are functions and objects. ABI resilience +is a high priority of Swift. + +- functions +- objects and their types + +We have to ask about all the + +Notes from meeting. +------------------- + +We definitely want to support resilient value types. Use cases: strings, +dates, opaque numbers, etc. Want to lock down API even without a +concrete implementation yet. + +This implies that we have to support runtime data layout. Need examples +of that. + +We do need to be resilient against adding \[unchanging\]. Okay to have +two levels of this: \[born\_unchanging\] for things that are universally +non-resilient, \[unchanging\] for things that were once resilient. +Proposed names: \[born\_fragile\] and \[fragile\]. + +Global functions always export a maximally-resilient entrypoint. If +there exist any \[fragile\] arguments, and there do not exist any +resilient arguments, they also export a \[fragile\] copy. Callers do… +something? Have to know what they're deploying against, I guess. + +Want some concrete representation for \[ref\] arguments. + +Notes from whiteboard conversation with Doug. +--------------------------------------------- + +What does fragility mean for different kinds of objects? + +structs (value types) - layout is fixed + +their fields - can access field directly rather than by getter/setter + +their methods - like any function + +classes (reference types) - layout of this struct component is fixed + +their fields - access directly vs. getter/setter + +their methods - like any function + +class extensions - like classes. what to do about layout of multiple +class extensions? very marginal + +functions - inlinable + +global variables - can be directly accessed. Type is born\_fragile: +value is global address. Type is resilient: lvalue is load of global +pointer. Type is fragile: value is load of global pointer, also optional +global address using same mechanism as global functions with fragile +argument types + +protocols - born\_fragile => laid out as vtable. Can these be +resilient? + +their implementations: contents of vtable are knowable + +enums - layout, set of variants + +Notes from second meeting +------------------------- + +Resilience attributes: + +- born\_fragile, fragile, resilient +- want to call born\_fragile => fragile, fragile + => fragile(macosx10.42) +- good except "default", more minimal thing is the more + aggressive thing. More important to have an ABI-checking tool +- use availability attributes scheme: platformX.Y.Z + +Components: Very much a build-system thing. + +Users should be able to write swift \[foo.swift\]+ and have it build an +executable. + +For anything more complicated, need a component description file. + +- hierarchy of components +- type of build product: executable, dylib, even object file +- non-Swift sources (object files, C files, whatever) +- deployment options (deploying on macosxX.Y.Z) +- need some sort of "include this subcomponent" declaration +- probably want some level of metaprogramming, maybe the preprocessor? + +Host of ways of find the component description file automatically: name +a directory (and find with fixed name), name nothing (and find in +current directory) + +Component organization is the seat of the decision algorithm for whether +we can access something resilient fragilely or not. + +- not necessarily just "are you in my component"; maybe "are you in my + domain/component tree/whatever" + +Resilience is lexically inherited. + +- Declarations inside a fragile enum are implicitly fragile, etc. +- Except anything inside a function is fragile. + +Break it down by types of declarations. + +- typealias has no resilience +- struct — the set/order of fields can change — means size/alignment, + layout, copy/destruction semantics, etc. can all change +- fields - direct access vs. getter/setter +- funcs - as if top level +- types - as if top level +- class — same as a structs, plus +- base classes — can't completely remove a base class (breaks + interface), but can introduce a new intermediate base +- virtual dispatch — table vs. dictionary, devirtualization (to + which decl?). Some amount of table lookup can be done as static vs. + dynamic offsets +- funcs — inlineability +- vars — direct access vs. getter/setter. Direct accesses for types + that aren't inherently fragile need to be indirected because they + may need to be dynamically allocated. In general, might be + actor-local, this is for when the model does say "global variable". +- extensions of classes — like class. Fields are always side-allocated + if we're extending a class not defined in this component + (w/i domain?). Making a class fragile is also a promise not to add + more fields in extensions in this component; probably need a way to + force a side-table. +- protocols — can't remove/change existing methods, but can add + defaulted methods. Doing this resiliently requires + load-time checking. vtable for non-defaulted methods, ? for rest? +- enum - set of directly represented cases +- enum elements - directly represented vs. injection/projection. +- enum - called out so that we can have an extensible thing that + promises no data fields. Always an i32 when resilient. +- const - fragile by default, as if a var otherwise diff --git a/docs/archive/Resilience.rst b/docs/archive/Resilience.rst deleted file mode 100644 index 84ec5f6d8ec19..0000000000000 --- a/docs/archive/Resilience.rst +++ /dev/null @@ -1,657 +0,0 @@ -:orphan: - -.. @raise litre.TestsAreMissing -.. _Resilience: - -Resilience -========== - -.. warning:: This is a very early design document discussing the feature of - Resilience. It should not be taken as a plan of record. - -Introduction ------------- - -One of Swift's primary design goals is to allow efficient execution of code -without sacrificing load-time abstraction of implementation. - -Abstraction of implementation means that code correctly written against a -published interface will correctly function when the underlying implementation -changes to anything which still satisfies the original interface. There are many -potential reasons to provide this sort of abstraction. Apple's primary interest -is in making it easy and painless for our internal and external developers to -improve the ecosystem of Apple products by creating good and secure programs and -libraries; subtle deployment problems and/or unnecessary dependencies on the -behavior of our implementations would work against these goals. - -Almost all languages provide some amount of abstraction of implementation. For -example, functions are usually opaque data types which fully abstract away the -exact sequence of operations performed. Similarly, adding a new field to a C -struct does not break programs which refer to a different field — those programs -may need to be recompiled, but once recompiled, they should continue to -work. (This would not necessarily be true if, say, fields were accessed by index -rather than by name.) - -Components ----------- - -Programs and libraries are not distributed as a legion of source files assembled -independently by the end user. Instead, they are packaged into larger components -which are distributed and loaded as a unit. Each component that comprises a -program may depend on some number of other components; this graph of -dependencies can be assumed to be acyclic. - -Because a component is distributed as a unit, ABI resilience within the -component is not required. It may still help to serve as a build- time -optimization, but Swift aims to offer substantially better build times than -C/C++ programs due to other properties of the language (the module system, the -lack of a preprocessor, the instantiation model, etc.). - -Components may be defined as broadly as an entire operating system or as -narrowly as the build products of an individual team. The development process of -a very large product (like an operating system) may discourage a model which -requires almost the entire system to be recompiled when a low-level component is -changed, even if recompilation is relatively fast. Swift aims to reduce the need -for unnecessary abstraction penalties where possible, but to allow essentially -any model that conforms to the basic rule of acyclicity. - -Abstraction Throughout the Program Lifecycle --------------------------------------------- - -Most languages provide different amounts of abstraction at different stages in -the lifecycle of a program. The stages are: - -1. Compilation, when individual source files are translated into a form suitable - for later phases. The abstractions of C/C++ object layout and C++ virtual - table layout are only provided until this stage; adding a field to a struct - or a virtual method to a class potentially require all users of those - interfaces to be recompiled. - -2. Bundling, when compiled source files are combined into a larger component; - this may occur in several stages, e.g. when object files are linked into a - shared library which is then bundled up into an OS distribution. Objective-C - classes in the non-fragile ABI are laid out by special processing that occurs - during bundling; this is done just as an optimization and the runtime can - force additional layout changes at execution time, but if that were not true, - the abstraction would be lost at this stage. - -3. Installation, when bundled components are placed on the system where they - will be executed. This is generally a good stage in which to lose abstraction - if implementations are always installed before interfaces. This is the first - stage whose performance is visible to the end user, but unless installation - times are absurdly long, there is little harm to doing extra processing here. - -4. Loading, when a component is loaded into memory in preparation for executing - it. Objective-C performs class layout at this stage; after loading completes, - no further changes to class layout are possible. Java also performs layout - and additional validation of class files during program loading. Processing - performed in this stage delays the start-up of a program (or plugin or other - component). - -5. Execution, when a piece of code actually evaluates. Objective-C guarantees - method lookup until this "stage", meaning that it is always possible to - dynamically add methods to a class. Languages which permit code to be - dynamically reloaded must also enforce abstraction during execution, although - JIT techniques can mitigate some of this expense. Processing performed in - this stage directly slows down the operation. - -Expressivity ------------- - -Certain language capabilities are affected by the choice of when to -break down the abstraction. - -* If the language runtime offers functions to dynamically explore and, by - reflection, use the language structures in a component, some amount of - metadata must survive until execution time. For example, invoking a function - would require the signature and ABI information about the types it uses to be - preserved sufficiently for the invocation code to reassemble it. - -* If the language runtime offers functions to dynamically change or augment the - language structures in a component, the full abstractions associated with - those structures must be preserved until execution time. For example, if an - existing "virtual" method can be replaced at runtime, no devirtualization is - permitted at compile time and there must be some way to at least map that - method to a vtable offset in all derived classes; and if such a method can be - dynamically added, there must be some ability for method dispatch to fall back - on a dictionary. - -* The above is equally true for class extensions in components which may be - loaded or unloaded dynamically. - -Performance ------------ - -The performance costs of abstraction mostly take three forms: - -* the direct cost of many small added indirections - -* code-size inflation due to the extra logic to implement these operations - -* diminished potential for more powerful optimizations such as inlining and code - motion - -As mentioned before, we can avoid these costs within components because classes -cannot be extended within components. - -We wish to avoid these costs wherever possible by exploiting the deployment -properties of programs. - -Summary of Design ------------------ - -Our current design in Swift is to provide opt-out load-time abstraction of -implementation for all language features. Alone, this would either incur -unacceptable cost or force widespread opting-out of abstraction. We intend to -mitigate this primarily by designing the language and its implementation to -minimize unnecessary and unintended abstraction: - -* Within the component that defines a language structure, all the details of its - implementation are available. - -* When language structures are not exposed outside their defining components, - their implementation is not constrained. - -* By default, language structures are not exposed outside their defining - components. This is independently desirable to reduce accidental API surface - area, but happens to also interact well with the performance design. - -* Avoiding unnecessary language guarantees and taking advantage of that - flexibility to limit load-time costs. - -We also intend to provide tools to detect inadvertent changes in -interfaces. - -Components ----------- - -(This is just a sketch and deserves its own design document.) - -Swift will have an integrated build system. This serves several purposes: - -* it creates a "single source of truth" about the project that can be shared - between tools, - -* it speeds up compiles by limiting redundant computation between files, and - -* it gives the compiler information about the boundaries between components. - -In complex cases, the build process is going to need to be described. Complex -cases include: - -* complex component hierarchies (see below) - -* the presence of non-Swift source files (to support directly: .s, .c, .o, maybe - .m, .mm, .cpp) - -* a build product other than an executable (to support directly: executable, - .dylib (.framework?), .o, maybe some binary component distribution) - -* library requirements - -* deployment requirements - -* compilation options more complicated than -On - -This specification file will basically function as the driver interface to Swift -and will probably need a similar host of features, e.g. QA overrides, -inheritance of settings from B&I. Some sort of target-based programming may also -be required. - -Components may be broken down into hierarchies of subcomponents. The component -graph must still be acyclic. - -Every component has a resilience domain , a component (either itself or an -ancestor in its component hierarchy) outside of which resilience is required. By -default, this is the top-level component in its hierarchy. - -Access ------- - -(sketch) - -A lot of code is not intended for use outside the component it appears in. Here -are four levels of access control, along with their proposed spellings: - -* [api] accessible from other components - -* [public] accessible only from this component (may need finer grades of control - to deal with non-trivial component hierarchies, e.g. public(somecomponent)) - -* [private] accessible only from this source file - -* [local] accessible only from things lexically included in the containing - declaration (may not be useful) - -A language structure's accessibility is inherited by default from its lexical -context. - -The global context (i.e. the default accessibility) is [public], i.e. -accessible from this component but not outside it. - -A language structure which is accessible outside the component it appears in is -said to be exported. - -Resilience ----------- - -In general, resilience is the ability to change the implementation of a language -structure without requiring further pre-load-time processing of code that uses -that structure and whose resilience domain does not include the component -defining that structure. - -Resilience does not permit changes to the language structure's interface. This -is a fairly vague standard (that will be spelled out below), but in general, an -interface change is a change which would cause existing code using that -structure to not compile or to compile using different formal types. - -Language structures may opt out of resilience with an attribute, -[fragile]. Deployment versions may be associated with the attribute, like so: -[fragile(macosx10.7, ios5)]. It is an interface change to remove an [fragile] -attribute, whether versioned or unversioned. It is an interface change to add an -unversioned [fragile] attribute. It is not an interface change to add a -versioned [fragile] attribute. There is also a [resilient] attribute, exclusive -to any form of [fragile], to explicitly describe a declaration as resilient. - -Resilience is lexically inherited. It is not lexically constrained; a resilient -language structure may have fragile sub-structures and vice- versa. The global -context is resilient, although since it is also [public] (and not [api]), -objects are not in practice constrained by resilience. - -We intend to provide a tool to automatically detect interface changes. - -Properties of types -------------------- - -A deployment is an abstract platform name and version. - -A type exists on a deployment if: - -* it is a builtin type, or - -* it is a function type and its argument and result types exist on the - deployment, or - -* it is a tuple type and all of its component types exist on the deployment, or - -* it is a struct, class, or enum type and it does not have an [available] - attribute with a later version for a matching platform name. - -It is an interface change for an exported type to gain an [available] attribute. - -A type is empty if it has a fragile representation (defined below) and: - -* it is a tuple type with no non-empty component types, or - -* it is a struct type with no non-empty fields, or - -* it is an enum type with one alternative which either carries no data or - carries data with an empty type. - -A type has a fragile representation if: - -* it is a builtin type. The representation should be obvious from the type. - -* it is a function type. The representation is a pair of two pointers: a valid - function pointer, and a potentially null retainable pointer. See the section - on calls for more information. - -* it is a tuple type with only fragilely-represented component types. The - representation of a tuple uses the Swift struct layout algorithm. This is - true even if the tuple does not have a fragile representation. - -* it is a class type (that is, a reference struct type). The representation is a - valid retainable pointer. - -* it is a fragile struct type with no resilient fields and no fields whose type - is fragilely represented. The representation uses the Swift struct layout - algorithm. - -A type has a universally fragile representation if there is no deployment of the -target platform for which the type exists and is not fragilely represented. It -is a theorem that all components agree on whether a type has a universal fragile -representation and, if so, what the size, unpadded size, and alignment of that -type is. - -Swift's struct layout algorithm takes as input a list of fields, and does the -following: - -1. The fields are ranked: - - * The universally fragile fields rank higher than the others. - - * If two fields A and B are both universally fragile, - - * If no other condition applies, fields that appear earlier in the original - sequence have higher rank. - -2. The size of the structure is initially 0. - - representations and A's type is more aligned than B's type, or otherwise if A - appears before B in the original ordering. - -3. Otherwise. Field A is ranked higher than Field B if - - * A has a universal fragile representation and B does not, or - -Swift provides the following types: - - -A language structure may be resilient but still define or have a type - -In the following discussion, it will be important to distinguish between types -whose values have a known representation and those which may not. - -Swift provides - -For some structures, it may be important to know that the structure has never -been deployed resiliently, so in general it is considered an interface change to -change a - -Resilience affects pretty much every language feature. - -Execution-time abstraction does not come without cost, and we do not wish to -incur those costs where unnecessary. Many forms of execution- time abstraction -are unnecessary except as a build-time optimization, because in practice the -software is deployed in large chunks that can be compiled at the same -time. Within such a resilience unit , many execution-time abstractions can be -broken down. However, this means that the resilience of a language structure is -context-dependent: it may need to be accessed in a resilient manner from one -resilience unit, but can be accessed more efficiently from another. A structure -which is not accessible outside its resilience unit is an important exception. A -structure is said to be exported if it is accessible in some theoretical context -outside its resilience unit. - -A structure is said to be resilient if accesses to it rely only on its - -A structure is said to be universally non-resilient if it is non- resilient in -all contexts in which it is accessible. - -Many APIs are willing to selectively "lock down" some of their component -structures, generally because either the form of the structure is inherent (like -a point being a pair of doubles) or important enough to performance to be worth -committing to (such as the accessors of a basic data structure). This requires -an [unchanging] annotation and is equivalent to saying that the structure is -universally non-resilient. - -Most language structures in Swift have resilience implications. This document -will need to be updated as language structures evolve and are enhanced. - -Type categories ---------------- - -For the purposes of this document, there are five categories of type in Swift. - -**Primitive types**: i1, float32, etc. Nominal types defined by the -implementation. - -**Functions**: () -> int, (NSRect, bool) -> (int, int), etc. Structural types -with - -**Tuples**: (NSRect, bool), (int, int), etc. Structural product types. - -**Named value types**: int, NSRect, etc. Nominal types created by a variety of -language structures. - -**Named reference types**: MyBinaryTree, NSWindow, etc. Nominal types created by -a variety of language structures. - -Primitive types are universally non-resilient. - -Function types are universally non-resilient (but see the section on calls). - -Tuple types are non-resilient if and only if all their component types are -non-resilient. - -Named types declared within a function are universally non-resilient. - -Named types with the [unchanging] annotation are universally non- -resilient. Problem, because of the need/desire to make things depend on whether -a type is universally non-resilient. Makes it impossible to add [unchanging] -without breaking ABI. See the call section. - -All other named types are non-resilient only in contexts that are in the same -resilient unit as their declaring file. - -Storage -------- - -Primitive types always have the primitive's size and alignment. - -Named reference types always have the size and alignment of a single pointer. - -Function types always have the size and alignment of two pointers, the first -being a maximally-nonresilient function pointer (see the section on calls) and -the second being a retain/released pointer. - -If a tuple type is not universally non-resilient, its elements are stored -sequentially using C struct layout rules. Layout must be computed at -runtime. Separate storage is not really a feasible alternative. - -Named types ------------ - -It is an error to place the [unchanging] annotation on any of these types: - -* a struct type with member types that are not universally non- resilient - -* an enum type with an enumerator whose type is not universally non- resilient - -* a class extension - -* a class whose primary definition contains a field which is not universally - non-resilient - -Classes -------- - -It is an error to place the [unchanging] annotation on a class extension. - -It is an error to place the [unchanging] annotation on a class whose primary -definition contains a field whose type is potentially resilient in a context -where the class is accessible. That is, if the class is exported, all of its -fields must be universally non- resilient. If it is not exported, all of its -fields must be non- resilient within its resilience unit. - -It is allowed to add fields to an [unchanging] class in a class extension. Such -fields are always side-stored, even if they are declared within the same -resilience unit. - -Objects -------- - -Right now, all types in Swift are "first-class", meaning that there is a broad -range of generic operations can be - -1. the size and layout of first-class objects: - - * local variables - - * global variables - - * dynamically*allocated objects - - * member sub*objects of a structure - - * base sub*objects of a class - - * element sub*objects of an array - - * parameters of functions - - * results of functions - -2. the set of operations on an object: - - * across all protocols - - * for a particular protocol (?) - -3. the set of operations on an object - - * ... - -ABI resilience means not making assumptions about language entities which limit -the ability of the maintainers of those entities to change them later. Language -entities are functions and objects. ABI resilience is a high priority of Swift. - -* functions - -* objects and their types - -We have to ask about all the - -Notes from meeting. -------------------- - -We definitely want to support resilient value types. Use cases: strings, dates, -opaque numbers, etc. Want to lock down API even without a concrete -implementation yet. - -This implies that we have to support runtime data layout. Need examples of that. - -We do need to be resilient against adding [unchanging]. Okay to have two levels -of this: [born_unchanging] for things that are universally non-resilient, -[unchanging] for things that were once resilient. Proposed names: -[born_fragile] and [fragile]. - -Global functions always export a maximally-resilient entrypoint. If there exist -any [fragile] arguments, and there do not exist any resilient arguments, they -also export a [fragile] copy. Callers do… something? Have to know what they're -deploying against, I guess. - -Want some concrete representation for [ref] arguments. - - - - -Notes from whiteboard conversation with Doug. ---------------------------------------------- - -What does fragility mean for different kinds of objects? - -structs (value types) - layout is fixed - -their fields - can access field directly rather than by getter/setter - -their methods - like any function - -classes (reference types) - layout of this struct component is fixed - -their fields - access directly vs. getter/setter - -their methods - like any function - -class extensions - like classes. what to do about layout of multiple class -extensions? very marginal - -functions - inlinable - -global variables - can be directly accessed. Type is born_fragile: value is -global address. Type is resilient: lvalue is load of global pointer. Type is -fragile: value is load of global pointer, also optional global address using -same mechanism as global functions with fragile argument types - -protocols - born_fragile => laid out as vtable. Can these be resilient? - -their implementations: contents of vtable are knowable - -enums - layout, set of variants - -Notes from second meeting -------------------------- - -Resilience attributes: - -* born_fragile, fragile, resilient - -* want to call born_fragile => fragile, fragile => fragile(macosx10.42) - -* good except "default", more minimal thing is the more aggressive thing. More - important to have an ABI-checking tool - -* use availability attributes scheme: platformX.Y.Z - -Components: Very much a build-system thing. - -Users should be able to write swift [foo.swift]+ and have it build an -executable. - -For anything more complicated, need a component description file. - -* hierarchy of components - -* type of build product: executable, dylib, even object file - -* non-Swift sources (object files, C files, whatever) - -* deployment options (deploying on macosxX.Y.Z) - -* need some sort of "include this subcomponent" declaration - -* probably want some level of metaprogramming, maybe the preprocessor? - -Host of ways of find the component description file automatically: name a -directory (and find with fixed name), name nothing (and find in current -directory) - -Component organization is the seat of the decision algorithm for whether we can -access something resilient fragilely or not. - -* not necessarily just "are you in my component"; maybe "are you in my - domain/component tree/whatever" - -Resilience is lexically inherited. - -* Declarations inside a fragile enum are implicitly fragile, etc. - -* Except anything inside a function is fragile. - -Break it down by types of declarations. - -* typealias has no resilience - -* struct — the set/order of fields can change — means size/alignment, layout, - copy/destruction semantics, etc. can all change - -* fields - direct access vs. getter/setter - -* funcs - as if top level - -* types - as if top level - -* class — same as a structs, plus - -* base classes — can't completely remove a base class (breaks interface), but - can introduce a new intermediate base - -* virtual dispatch — table vs. dictionary, devirtualization (to which - decl?). Some amount of table lookup can be done as static vs. dynamic offsets - -* funcs — inlineability - -* vars — direct access vs. getter/setter. Direct accesses for types that aren't - inherently fragile need to be indirected because they may need to be - dynamically allocated. In general, might be actor-local, this is for when the - model does say "global variable". - -* extensions of classes — like class. Fields are always side-allocated if we're - extending a class not defined in this component (w/i domain?). Making a class - fragile is also a promise not to add more fields in extensions in this - component; probably need a way to force a side-table. - -* protocols — can't remove/change existing methods, but can add defaulted - methods. Doing this resiliently requires load-time checking. vtable for - non-defaulted methods, ? for rest? - -* enum - set of directly represented cases - -* enum elements - directly represented vs. injection/projection. - -* enum - called out so that we can have an extensible thing that promises no - data fields. Always an i32 when resilient. - -* const - fragile by default, as if a var otherwise - - - diff --git a/docs/contents.md b/docs/contents.md new file mode 100644 index 0000000000000..1a8e17760786d --- /dev/null +++ b/docs/contents.md @@ -0,0 +1,2 @@ +Contents +======== diff --git a/docs/contents.rst b/docs/contents.rst deleted file mode 100644 index 38b04b3fa6aa9..0000000000000 --- a/docs/contents.rst +++ /dev/null @@ -1,24 +0,0 @@ -.. @raise litre.TestsAreMissing -.. _contents: - -Contents -======== - -.. toctree:: - :maxdepth: 1 - - IndexInvalidation - AccessControl - DriverInternals - DriverParseableOutput - ErrorHandling - ErrorHandlingRationale - Generics - LogicalObjects - ObjectInitialization - Pattern Matching - StoredAndComputedVariables - SIL - TypeChecker - DebuggingTheCompiler - diff --git a/docs/proposals/Accessors.md b/docs/proposals/Accessors.md new file mode 100644 index 0000000000000..a186182838451 --- /dev/null +++ b/docs/proposals/Accessors.md @@ -0,0 +1,1289 @@ +Optimizing Accessors in Swift +============================= + +Definitions +----------- + +An abstract storage declaration is a language construct that declares a +means of accessing some sort of abstract entity. I'll just say "storage" +hereafter. + +Swift provides three storage declarations: + +- a single named entity, called a *variable* and declared with `var` +- a single named entity which can never be reassigned, called a + *constant* and declared with `let` +- a compound unnamed entity accessed with an index, called a + *subscript* and declared with `subscript` + +These features are similar to those in other languages. Swift notably +lacks compound named entities, such as C\#'s indexed properties; the +design team intentionally chose to favor the use of named single +entities of subscriptable type. + +It's useful to lump the two kinds of single named entities together; +I'll just call them both "variables" hereafter. + +Subscripts must always be instance type members. When a variable is a +type member, it's called a *property*. + +When I say that these entities are *abstract*, I mean that they're not +directly tied to any particular implementation. All of them may be +backed directly by memory storing values of the entity's type, or they +may simply provide a way of invoking arbitrary code to access a logical +memory, or they may be a little of both. + +Full-value accesses +------------------- + +All accesses to storage, no matter the implementation, can be performed +with two primitive operations: + +- a full-value load, which creates a new copy of the current value +- a full-value store, which overwrites the current value with a + different, independent value + +A function which implements a full-value load is called a *getter*; a +full-value store, a *setter*. + +An operation which calls for a full-value load into a temporary, then a +modification of the temporary, then a full-value store of the temporary +into the original entity, is called a *write-back*. + +Implementing accesses with full-value accesses introduces two problems: +subobject clobbering and performance. + +### Subobject clobbering + +Subobject clobbering is a semantic issue. It occurs when there are two +changes to an entity in flight at once, as in: + + swap(&point.x, &point.y) + +(By "at once", I mean synchronously. Unlike Java, Swift is willing to +say that an *asynchronous* simultaneous access (mutating or not) to an +entity that's being modified is completely undefined behavior, with any +sort of failure permitted, up to and including memory corruption and +crashes. As Swift transitions towards being a systems language, it can +selectively choose to define some of this behavior, e.g. allowing racing +accesses to different elements of an array.) + +Subobject clobbering is a problem with user expectation. Users are +unlikely to write code that intentionally modifies two obviously +overlapping entities, but they might very reasonably write code that +modifies two "separate" sub-entities. For example, they might write code +to swap two properties of a struct, or two elements of an array. The +natural expansion of these subobject accesses using whole-object +accesses generates code that "loses" the changes made to one of the +objects: + + var point0 = point + var x = point0.x + var point1 = point + var y = point1.y + swap(&x, &y) + point1.y = y + point = point1 + point0.x = x + point = point0 + +Note that `point.y` is left unchanged. + +#### Local analysis + +There are two straightforward solutions to subobject clobbering, both +reliant on doing local analysis to recognize two accesses which +obviously alias (like the two references to `point` in the above +example). Once you've done this, you can either: + +- Emit an error, outlawing such code. This is what Swift currently + does (but only when the aliasing access must be implemented with + full-value loads and stores). +- Use a single temporary, potentially changing the semantics. + +It's impossible to guarantee the absence of subobject clobbering with +this analysis without extremely heavy-handed languages changes. +Fortunately, subobject clobbering is "only" a semantics issue, not a +memory-safety issue, at least as long as it's implemented with +full-value accesses. + +#### Reprojection + +A more general solution is to re-project the modified subobject from +scratch before writing it back. That is, you first acquire an initial +value like normal for the subobject you wish to modify, then modify that +temporary copy in-place. But instead of then recursively consuming all +the intermediate temporaries when writing them back, you drop them all +and recompute the current value, then write the modified subobject to +it, then write back all the way up. That is: + + var point0 = point + var x = point0.x + var point1 = point + var y = point1.y + swap(&x, &y) + point1 = point // reload point1 + point1.y = y + point = point1 + point0 = point // reload point0 + point0.x = x + point = point0 + +In this example, I've applied the solution consistently to all accesses, +which protects against unseen modifications (e.g. during the call to +`swap`) at the cost of performing two extra full-value loads. + +You can heuristically lower this by combining it with a simple local +analysis and only re-projecting when writing back to other l-values +besides the last. In other words, generate code that will work as long +as the entity is not modified behind abstraction barriers or through +unexpected aliases: + + var point0 = point + var x = point0.x + var point1 = point + var y = point1.y + swap(&x, &y) + point1.y = y // do not reload point1 + point = point1 + point0 = point // reload point0 + point0.x = x + point = point0 + +Note that, in either solution, you've introduced extra full-value loads. +This may be quite expensive, and it's not guaranteed to be semantically +equivalent. + +### Performance + +There are three major reasons why full-value accesses are inefficient. + +#### Unnecessary subobject accesses + +The first is that they may load or store more than is necessary. + +As an obvious example, imagine a variable of type `(Int,Int)`; even if +my code only accesses the first element of the tuple, full-value +accesses force me to read or write the second element as well. That +means that, even if I'm purely overwriting the first element, I actually +have to perform a full-value load first so that I know what value to use +for the second element when performing the full-value store. + +Additionally, while unnecessarily loading the second element of an +`(Int,Int)` pair might seem trivial, consider that the tuple could +actually have twenty elements, or that the second element might be +non-trivial to copy (e.g. if it's a retainable pointer). + +#### Abstraction barriers + +A full-value load or store which you can completely reason about is one +thing, but if it has to be performed as a call, it can be a major +performance drag. + +For one, calls do carry a significant amount of low-level overhead. + +For another, optimizers must be extremely conservative about what a call +might do. A retainable pointer might have to be retained and later +released purely to protect against the possibility that a getter might, +somehow, cause the pointer to otherwise be deallocated. + +Furthermore, the conventions of the call might restrict performance. One +way or another, a getter for a retainable pointer generally returns at ++1, meaning that as part of the return, it is retained, forcing the +caller to later release. If the access were instead direct to memory, +this retain might be avoidable, depending on what the caller does with +the pointer. + +#### Copy-on-write + +These problems are compounded by copy-on-write (COW) types. In Swift, a +copy-on-write value embeds an object reference. Copying the value has +low immediate cost, because it simply retains the existing reference. +However, modifying a value requires the reference to be made unique, +generally by copying the data held by the value into a fresh object. +I'll call this operation a *structural copy* in an effort to avoid the +more treacherous term "deep copy". + +COW types are problematic with full-value accesses for several reasons. + +First, COW types are often used to implement aggregates and thus often +have several distinguishable subobjects which users are likely to think +of as independent. This heightens the dangers of subobject clobbering. + +Second, a full-value load of a COW type implies making the object +reference non-unique. Changing the value at this point will force a +structural copy. This means that modifying a temporary copy has +dramatically worse performance compared to modifying the original entity +in-place. For example: + + window.name += " (closing)" + +If `&window.name` can be passed directly to the operator, and the string +buffer is uniquely referenced by that string, then this operation may be +as cheap as copying a few characters into the tail of the buffer. But if +this must be done with a write-back, then the temporary will never have +a unique reference, and there will always be an unneeded structural +copy. + +Conservative access patterns +---------------------------- + +When you know how storage is implemented, it's straightforward to +generate an optimal access to it. There are several major reasons why +you might not know how a storage declaration is implemented, though: + +- It might be an abstract declaration, not a concrete declaration. + Currently this means a protocol member, but Swift may someday add + abstract class members. +- It might be a non-final class member, where the implementation you + can see is potentially overridable by a subclass. +- It might be a resilient declaration, where you know only that the + entity exists and know nothing statically about its implementation. + +In all of these cases, you must generate code that will handle the worst +possible case, which is that the entity is implemented with a getter and +a setter. Therefore, the conservative access pattern includes opaque +getter and setter functions. + +However, for all the reasons discussed above, using unnecessary +full-value accesses can be terrible for performance. It's really bad if +a little conservatism --- e.g. because Swift failed to devirtualize a +property access --- causes asymptotic inefficiencies. Therefore, Swift's +native conservative access pattern also includes a third accessor which +permits direct access to storage when possible. This accessor is called +`materializeForSet`. + +`materializeForSet` receives an extra argument, which is an +uninitialized buffer of the value type, and it returns a pointer and a +flag. When it can provide direct access to storage for the entity, it +constructs a pointer to the storage and returns false. When it can't, it +performs a full-value load into the buffer and returns true. The caller +performs the modification in-place on the returned pointer and then, if +the flag is true, passes the value to the setter. + +The overall effect is to enable direct storage access as a dynamic +optimization when it's impossible as a static optimization. + +For now, `materializeForSet` is always automatically generated based on +whether the entity is implemented with a computed setter. It is possible +to imagine data structures that would benefit from having this lifted to +a user-definable feature; for example, a data structure which sometimes +holds its elements in memory but sometimes does not. + +`materializeForSet` can provide direct access whenever an address for +the storage can be derived. This includes when the storage is +implemented with a `mutableAddress` accessor, as covered below. +Observing accessors currently prevent `materializeForSet` from offering +direct access; that's fixable for `didSet` using a slightly different +code pattern, but `willSet` is an inherent obstacle. + +Independent of any of the other optimizations discussed in this +whitepaper, `materializeForSet` had the potential to immediately +optimize the extremely important case of mutations to COW values in +un-devirtualized class properties, with fairly minimal risk. Therefore, +`materializeForSet` was implemented first, and it shipped in Xcode 6.1. + +Direct access at computed addresses +----------------------------------- + +What entities can be directly accessed in memory? Non-computed variables +make up an extremely important set of cases; Swift has enough built-in +knowledge to know that it can provide direct access to them. But there +are a number of other important cases where the address of an entity is +not built-in to the compiler, but where direct access is nonetheless +possible. For example, elements of a simple array always have +independent storage in memory. Most benchmarks on arrays would profit +from being able to modify array elements in-place. + +There's a long chain of proposals in this area, many of which are +refinement on previous proposals. None of these proposals has yet +shipped in Xcode. + +### Addressors + +For something like a simple array (or any similar structure, like a +deque) which is always backed by a buffer, it makes sense for the +implementor to simply define accessors which return the address of the +element. Such accessors are called *addressors*, and there are two: +`address` and `mutableAddress`. + +The conservative access pattern can be generated very easily from this: +the getter calls `address` and loads from it, the setter calls +`mutableAddress` and stores to it, and `materializeForSet` provides +direct access to the address returned from `mutableAddress`. + +If the entity has type `T`, then `address` returns an `UnsafePointer` +and `mutableAddress` returns an `UnsafeMutablePointer`. This means +that the formal type of the entity must exactly match the formal type of +the storage. Thus, the standard subscript on `Dictionary` cannot be +implemented using addressors, because the formal type of the entity is +`V?`, but the backing storage holds a `V`. (And this is in keeping with +user expectations about the data structure: assigning `nil` at a key is +supposed to erase any existing entry there, not create a new entry to +hold `nil`.) + +This simple addressor proposal was the first prong of our efforts to +optimize array element access. Unfortunately, while it is useful for +several other types (such as `ContiguousArray` and +`UnsafeMutablePointer`), it is not flexible enough for the `Array` type. + +### Mixed addressors + +Swift's chief `Array` type is only a simple array when it is not +interacting with Objective-C. Type bridging requires `Array` to be able +to store an immutable `NSArray` instance, and the `NSArray` interface +does not expose the details of how it stores elements. An `NSArray` is +even permitted to dynamically generate its values in its +`objectAtIndex:` method. And it would be absurd for `Array` to perform a +structural copy during a load just to make non-mutating accesses more +efficient! So the load access pattern for `Array`'s subscript +declaration must use a getter. + +Fortunately, this requirement does not preclude using an addressor for +mutating accesses. Mutations to `Array` always transition the array to a +unique contiguous buffer representation as their first step. This means +that the subscript operator can sensibly return an address when it's +used for the purposes of mutation: in other words, exactly when +`mutableAddress` would be invoked. + +Therefore, the second prong of our efforts to optimize array element +access was to allow entities to be implemented with the combination of a +`get` accessor and a `mutableAddress` accessor. This is straightforward +in the user model, where it simply means lifting a restriction. It's +more complex behind the scenes because it broke what was previously a +clean conceptual division between "physical" and "logical" l-values. + +Mixed addressors have now been adopted by `Array` to great success. As +expected, they substantially improved performance mutating COW array +elements. But they also fix an important instance of subobject +clobbering, because modifications to different subobjects (notably, +different elements of the same array) can occur simultaneously by simply +projecting out their addresses in the unique buffer. For example, this +means that it's possible to simply swap two elements of an array +directly: + + swap(&array[i], &array[j]) + + // Expanded: + array.transitionToUniquelyReferenced() + let address_i = array.buffer.storage + i + array.transitionToUniquelyReferenced() + let address_j = array.buffer.storage + j + swap(address_i, address_j) + +Mixed addressors weren't completely implemented until very close to the +Xcode 6.1 deadline, and they changed code-generation patterns enough to +break a number of important array-specific optimizations. Therefore, the +team sensibly decided that they were too risky for that release, and +that there wasn't enough benefit from other applications to justify +including any of the addressor work. + +In a way, that was a fortunate decision, because the naive version of +addressors implemented so far in Swift creates a safety hole which would +otherwise have been exposed to users. + +### Memory unsafety of addressors + +The semantics and memory safety of operations on COW types rely on a +pair of simple rules: + +- A non-mutating operation must own a reference to the buffer for the + full course of the read. +- A mutating operation must own a unique reference to the buffer for + the full course of the mutation. + +Both rules tend to be naturally satisfied by the way that operations are +organized into methods. A value must own a reference to its buffer at +the moment that a method is invoked on it. A mutating operation +immediately transitions the buffer to a unique reference, performing a +structural copy if necessary. This reference will remain valid for the +rest of the method as long as the method is *atomic*: as long as it does +not synchronously invoke arbitrary user code. + +(This is a single-threaded notion of atomicity. A second thread which +modifies the value simultaneously can clearly invalidate the assumption. +But that would necessarily be a data race, and the language design team +is willing to say that such races have fully undefined behavior, and +arbitrary consequences like memory corruption and crashes are acceptable +in their wake.) + +However, addressors are not atomic in this way: they return an address +to the caller, which may then interleave arbitrary code before +completing the operation. This can present the opportunity for +corruption if the interleaved code modifies the original value. Consider +the following code: + + func operate(inout value: Int, count: Int) { ... } + + var array: [Int] = [1,2,3,4] + operate(&array[0], { array = []; return 0 }()) + +The dynamic sequence of operations performed here will expand like so: + + var array: [Int] = [1,2,3,4] + let address = array.subscript.mutableAddress(0) + array = [] + operate(address, 0) + +The assignment to `array` within the closure will release the buffer +containing `address`, thus passing `operate` a dangling pointer. + +Nor can this be fixed with a purely local analysis; consider: + + class C { var array: [Int] } + let global_C = C() + + func assign(inout value: Int) { + C.array = [] + value = 0 + } + + assign(&global_C.array[0]) + +### Fixing the memory safety hole + +Conceptually, the correct fix is to guarantee that the rules are +satisfied by ensuring that the buffer is retained for the duration of +the operation. Any interleaving modifications will then see a +non-uniquely-referenced buffer and perform a structural copy: + + // Project the array element. + let address = array.subscript.mutableAddress(0) + + // Remember the new buffer value and keep it retained. + let newArrayBuffer = array.buffer + retain(newArrayBuffer) + + // Reassign the variable. + release(array.buffer) + array.buffer = ... + + // Perform the mutation. These changes will be silently lost, but + // they at least won't be using deallocated memory. + operate(address, 0) + + // Release the "new" buffer. + release(newArrayBuffer) + +Note that this still leaves a semantic hole if the original value is +copied in interleaving code before the modification, because the +subsequent modification will be reflected in the copy: + + // Project the array element. + let address = array.subscript.mutableAddress(0) + + // Remember the new buffer value and keep it retained. + let newArrayBuffer = array.buffer + retain(newArrayBuffer) + + // Copy the value. Note that arrayCopy uses the same buffer that + // 'address' points into. + let arrayCopy = array + retain(arrayCopy.buffer) + + // Perform the mutation. + operate(address, 0) + + // Release the "new" buffer. + release(newArrayBuffer) + +This might be unexpected behavior, but the language team is willing to +accept unexpected behavior for this code. What's non-negotiable is +breaking memory safety. + +Unfortunately, applying this fix naively reintroduces the problem of +subobject clobbering: since a modification of one subobject immediately +retains a buffer that's global to the entire value, an interleaved +modification of a different subobject will see a non-unique buffer +reference and therefore perform a structural copy. The modifications to +the first subobject will therefore be silently lost. + +Unlike the interleaving copy case, this is seen as unacceptable. +Notably, it breaks swapping two array elements: + + // Original: + swap(&array[i], &array[j]) + + // Expanded: + + // Project array[i]. + array.transitionToUniquelyReferenced() + let address_i = array.buffer.storage + i + let newArrayBuffer_i = array.buffer + retain(newArrayBuffer_i) + + // Project array[j]. Note that this transition is guaranteed + // to have to do a structural copy. + array.transitionToUniquelyReferenced() + let address_j = array.buffer.storage + j + let newArrayBuffer_j = array.buffer + retain(newArrayBuffer_j) + + // Perform the mutations. + swap(address_i, address_j) + + // Balance out the retains. + release(newArrayBuffer_j) + release(newArrayBuffer_i) + +Acceptability +------------- + +This whitepaper has mentioned several times that the language team is +prepared to accept such-and-such behavior but not prepared to accept +some other kind of behavior. Clearly, there is a policy at work. What is +it? + +### General philosophy + +For any given language problem, a perfect solution would be one which: + +- guarantees that all operations complete without crashing or + corrupting the program state, +- guarantees that all operations produce results according to + consistent, reliable, and intuitive rules, +- does not limit or require complex interactions with the remainder of + the language, and +- imposes no performance cost. + +These goals are, however, not all simultaneously achievable, and +different languages reach different balances. Swift's particular +philosophy is as follows: + +- The language should be as dynamically safe as possible. + Straightforward uses of ordinary language features may cause dynamic + failure, but the should never corrupt the program state. Any unsafe + language or library features (other than simply calling into C code) + should be explicitly labeled as unsafe. + + A dynamic failure should mean that the program reliably halts, + ideally with a message clearly describing the source of the failure. + In the future, the language may allow for emergency recovery from + such failures. + +- The language should sit on top of C, relying only on a relatively + unobtrusive runtime. Accordingly, the language's interactions with + C-based technologies should be efficient and obvious. +- The language should allow a static compiler to produce efficient + code without dynamic instrumentation. Accordingly, static analysis + should only be blocked by incomplete information when the code uses + an obviously abstract language feature (such as calling a class + method or an unknown function), and the language should provide + tools to allow programmers to limit such cases. + + (Dynamic instrumentation can, of course, still help, but it + shouldn't be required for excellent performance.) + +### General solutions + +A language generally has six tools for dealing with code it considers +undesireable. Some of this terminology is taken from existing standards, +others not. + +- The language may nonetheless take steps to ensure that the code + executes with a reliable result. Such code is said to have + *guaranteed behavior*. +- The language may report the code as erroneous before it executes. + Such code is said to be *ill formed*. +- The language may reliably report the code as having performed an + illegal operation when it executes. Such code is said to be + *asserting* or *aborting*. +- The language may allow the code to produce an + arbitrary-but-sound result. Such code is said to have *unspecified + behavior* or to have produced an *unspecified value*. +- The language may allow the code to produce an unsound result which + will result in another of these behaviors, but only if used. Such + code is said to have produced a *trap value*. +- The language may declare the code to be completely outside of the + guarantees of the language. Such code is said to have *undefined + behavior*. + +In keeping with its design philosophy, Swift has generally limited +itself to the first four solutions, with two significant exceptions. + +The first exception is that Swift provides several explicitly unsafe +language and library features, such as `UnsafePointer` and +`unowned(unsafe)`. The use of these features is generally subject to +undefined behavior rules. + +The second exception is that Swift does not make any guarantees about +programs in the presence of race conditions. It is extremely difficult +to make even weak statements about the behavior of a program with a race +condition without either: + +- heavily restricting shared mutable state on a language level, which + would require invasive changes to how the language interacts with C; +- forcing implicit synchronization when making any change to + potentially shared memory, which would cripple performance and + greatly complicate library implementation; or +- using a garbage collector to manage all accessible memory, which + would impose a very large burden on almost all of Swift's + language goals. + +Therefore, Swift does surrender safety in the presence of races. + +### Acceptability conditions for storage accesses + +Storage access involves a tension between four goals: + +- Preserving all changes when making simultaneous modifications to + distinct subobjects; in other words, avoiding subobject clobbering +- Performing a predictable and intuitive sequence of operations when + modifying storage that's implemented with a getter and setter +- Avoiding unnecessary copies of a value during a modification, + especially when this forces a structural copy of a COW value +- Avoiding memory safety holes when accessing storage that's been + implemented with memory. + +Reprojection\_ is good at preserving changes, but it introduces extra +copies, and it's less intuitive about how many times getters and setters +will be called. There may be a place for it anyway, if we're willing to +accept the extra conceptual complexity for computed storage, but it's +not a reasonable primary basis for optimizing the performance of storage +backed by memory. + +Solutions permitting in-place modification are more efficient, but they +do have the inherent disadvantage of having to vend the address of a +value before arbitrary interleaving code. Even if the address remains +valid, and the solution to that avoids subobject clobbering, there's an +unavoidable issue that the write can be lost because the address became +dissociated from the storage. For example, if your code passes +`&array[i]` to a function, you might plausibly argue that changes to +that argument should show up in the `i`th element of `array` even if you +completely reassign `array`. Reprojection\_ could make this work, but +in-place solutions cannot efficiently do so. So, for any in-place +solution to be acceptable, there does need to be some rule specifying +when it's okay to "lose track" of a change. + +Furthermore, the basic behavior of COW means that it's possible to copy +an array with an element under modification and end up sharing the same +buffer, so that the modification will be reflected in a value that was +technically copied beforehand. For example: + + var array = [1,2,3] + var oldArray : [Int] = [] + + // This function copies array before modifying it, but because that + // copy is of an value undergoing modification, the copy will use + // the same buffer and therefore observe updates to the element. + func foo(inout element: Int) { + oldArray = array + element = 4 + } + + // Therefore, oldArray[2] will be 4 after this call. + foo(&array[2]) + +Nor can this be fixed by temporarily moving the modified array aside, +because that would prevent simultaneous modifications to different +elements (and, in fact, likely cause them to assert). So the rule will +also have to allow this. + +However, both of these possibilities already come up in the design of +both the library and the optimizer. The optimizer makes a number of +assumptions about aliasing; for example, the general rule is that +storage bound to an `inout` parameter cannot be accessed through other +paths, and while the optimizer is not permitted to compromise memory +safety, it is permitted to introduce exactly this kind of unexpected +behavior where aliasing accesses may or may not the storage as a +consistent entity. + +#### Formal accesses + +That rule leads to an interesting generalization. Every modification of +storage occurs during a *formal access* (FA) to that storage. An FA is +also associated with zero or more *designated storage names* (DSNs), +which are `inout` arguments in particular execution records. An FA +arises from an l-value expression, and its duration and DSN set depend +on how the l-value is used: + +- An l-value which is simply loaded from creates an instantaneous FA + at the time of the load. The DSN set is empty. + + Example: + + foo(array) + // instantaneous FA reading array + +- An l-value which is assigned to with `=` creates an instantaneous FA + at the time of the primitive assignment. The DSN set is empty. + + Example: + + array = [] + // instantaneous FA assigning array + + Note that the primitive assignment strictly follows the evaluation + of both the l-value and r-value expressions of the assignment. For + example, the following code: + + // object is a variable of class type + object.array = object.array + [1,2,3] + + produces this sequence of formal accesses: + + // instantaneous FA reading object (in the left-hand side) + // instantaneous FA reading object (in the right-hand side) + // instantaneous FA reading object.array (in the right-hand side) + // evaluation of [1,2,3] + // evaluation of + + // instantaneous FA assigning object.array + +- An l-value which is passed as an `inout` argument to a call creates + an FA beginning immediately before the call and ending immediately + after the call. (This includes calls where an argument is implicitly + passed `inout`, such as to mutating methods or user-defined + assignment operators such as `+=` or `++`.) The DSN set contains the + `inout` argument within the call. + + Example: + + func swap(inout lhs: T, inout rhs: T) {} + + // object is a variable of class type + swap(&leftObject.array, &rightObject.array) + + This results in the following sequence of formal accesses: + + // instantaneous FA reading leftObject + // instantaneous FA reading rightObject + // begin FA for inout argument leftObject.array (DSN={lhs}) + // begin FA for inout argument rightObject.array (DSN={rhs}) + // evaluation of swap + // end FA for inout argument rightObject.array + // end FA for inout argument leftObject.array + +- An l-value which is used as the base of a member storage access + begins an FA whose duration is the same as the duration of the FA + for the subobject l-value. The DSN set is empty. + + Example: + + swap(&leftObject.array[i], &rightObject.array[j]) + + This results in the following sequence of formal accesses: + + // instantaneous FA reading leftObject + // instantaneous FA reading i + // instantaneous FA reading rightObject + // instantaneous FA reading j + // begin FA for subobject base leftObject.array (DSN={}) + // begin FA for inout argument leftObject.array[i] (DSN={lhs}) + // begin FA for subobject base rightObject.array (DSN={}) + // begin FA for inout argument rightObject.array[j] (DSN={rhs}) + // evaluation of swap + // end FA for subobject base rightObject.array[j] + // end FA for inout argument rightObject.array + // end FA for subobject base leftObject.array[i] + // end FA for inout argument leftObject.array + +- An l-value which is the base of an ! operator begins an FA whose + duration is the same the duration of the FA for the + resulting l-value. The DSN set is empty. + + Example: + + // left is a variable of type T + // right is a variable of type T? + swap(&left, &right!) + + This results in the following sequence of formal accesses: + + // begin FA for inout argument left (DSN={lhs}) + // begin FA for ! operand right (DSN={}) + // begin FA for inout argument right! (DSN={rhs}) + // evaluation of swap + // end FA for inout argument right! + // end FA for ! operand right + // end FA for inout argument left + +- An l-value which is the base of a ? operator begins an FA whose + duration begins during the formal evaluation of the l-value and ends + either immediately (if the operand was nil) or at the end of the + duration of the FA for the resulting l-value. In either case, the + DSN set is empty. + + Example: + + // left is a variable of optional struct type + // right is a variable of type Int + left?.member += right + + This results in the following sequence of formal accesses, assuming + that `left` contains a value: + + // begin FA for ? operand left (DSN={}) + // instataneous FA reading right (DSN={}) + // begin FA for inout argument left?.member (DSN={lhs}) + // evaluation of += + // end FA for inout argument left?.member + // end FA for ? operand left + +The FAs for all `inout` arguments to a call begin simultaneously at a +point strictly following the evaluation of all the argument expressions. +For example, in the call `foo(&array, array)`, the evaluation of the +second argument produces a defined value, because the FA for the first +argument does not begin until after all the arguments are formally +evaluated. No code should actually be emitted during the formal +evaluation of `&array`, but for an expression like +`someClassReference.someArray[i]`, the class r-value and index +expressions would be fully evaluated at that time, and then the l-value +would be kept abstract until the FA begins. Note that this requires +changes in SILGen's current code generation patterns. + +The FA rule for the chaining operator `?` is an exception to the +otherwise-simple intuition that formal accesses begin immediately before +the modification begins. This is necessary because the evaluation rules +for `?` may cause arbitrary computation to be short-circuited, and +therefore the operand must be accessed during the formal evaluation of +the l-value. There were three options here: + +- Abandon short-circuiting for assignments to optional l-values. This + is a very high price; short-circuiting fits into user intuitions + about the behavior of the chaining operator, and it can actually be + quite awkward to replicate with explicit accesses. +- Short-circuit using an instantaneous formal access, then start a + separate formal access before the actual modification. In other + words, evaluation of `X += Y` would proceed by first determining + whether `X` exists (capturing the results of any r-value + components), discarding any projection information derived from + that, evaluating `Y`, reprojecting `X` again (using the saved + r-value components and checking again for whether the l-value + exists), and finally calling the `+=` operator. + + If `X` involves any sort of computed storage, the steps required to + evaluate this might be... counter-intuitive. + +- Allow the formal access to begin during the formal evaluation of + the l-value. This means that code like the following will have + unspecified behavior: + + array[i]?.member = deriveNewValueFrom(array) + +In the end, I've gone with the third option. The intuitive explanation +is that `array` has to be accessed early in order to continue the +evaluation of the l-value. I think that's comprehensible to users, even +if it's not immediately obvious. + +#### Disjoint and non-disjoint formal accesses + +I'm almost ready to state the core rule about formal accesses, but first +I need to build up a few more definitions. + +An *abstract storage location* (ASL) is: + +- a global variable declaration; +- an `inout` parameter declaration, along with a reference to a + specific execution record for that function; +- a local variable declaration, along with a reference to a specific + execution record for that declaration statement; +- a static/class property declaration, along with a type having that + property; +- a struct/enum instance property declaration, along with an ASL for + the base; +- a struct/enum subscript declaration, along with a concrete index + value and an ASL for the base; +- a class instance property declaration, along with an instance of + that class; or +- a class instance subscript declaration, along with a concrete index + value and an instance of that class. + +Two abstract storage locations may be said to *overlap*. Overlap +corresponds to the imprecise intuition that a modification of one +location directly alters the value of another location. Overlap is an +"open" property of the language: every new declaration may introduce its +own overlap behavior. However, the language and library make certain +assertions about the overlap of some locations: + +- An `inout` parameter declaration overlaps exactly the set of ASLs + overlapped by the ASL which was passed as an argument. +- If two ASLs are both implemented with memory, then they overlap only + if they have the same kind in the above list and the corresponding + data match: + + > - execution records must represent the same execution + > - types must be the same + > - class instances must be the same + > - ASLs must overlap + +- For the purposes of the above rule, the subscript of a standard + library array type is implemented with memory, and the two indexes + match if they have the same integer value. +- For the purposes of the above rule, the subscript of a standard + library dictionary type is implemented with memory, and the two + indexes match if they compare equal with `==`. + +Because this definition is open, it is impossible to completely +statically or dynamically decided it. However, it would still be +possible to write a dynamic analysis which decided it for common +location kinds. Such a tool would be useful as part of, say, an +ASan-like dynamic tool to diagnose violations of the +unspecified-behavior rule below. + +The overlap rule is vague about computed storage partly because computed +storage can have non-obvious aliasing behavior and partly because the +subobject clobbering caused by the full-value accesses required by +computed storage can introduce unexpected results that can be reasonably +glossed as unspecified values. + +This notion of abstract storage location overlap can be applied to +formal accesses as well. Two FAs `x` and `y` are said to be *disjoint* +if: + +- they refer to non-overlapping abstract storage locations or +- they are the base FAs of two disjoint member storage accesses `x.a` + and `y.b`. + +Given these definitions, the core unspecified-behavior rule is: + +> If two non-disjoint FAs have intersecting durations, and neither FA is +> derived from a DSN for the other, then the program has unspecified +> behavior in the following way: if the second FA is a load, it yields +> an unspecified value; otherwise, both FAs store an unspecified value +> in the storage. + +Note that you cannot have two loads with intersecting durations, because +the FAs for loads are instantaneous. + +Non-overlapping subobject accesses make the base accesses disjoint +because otherwise code like `swap(&a[0], &a[1])` would have unspecified +behavior, because the two base FAs are to clearly overlapping locations +and have intersecting durations. + +Note that the optimizer's aliasing rule falls out from this rule. If +storage has been bound as an `inout` argument, accesses to it through +any path not derived from the `inout` argument will start a new FA for +overlapping storage, the duration of which will necessarily intersect +duration with that of the FA through which the `inout` argument was +bound, causing unspecified behavior. If the `inout` argument is +forwarded to another call, that will start a new FA which is validly +based on a DSN of the first; but an attempt to modify the storage +through the first `inout` argument while the second call is active will +create a third FA not based on the DSN from the second `inout` call, +causing a conflict there. Therefore a function may assume that it can +see all accesses to the storage bound to an `inout` argument. + +#### If you didn't catch all that... + +That may have been a somewhat intense description, so here's a simple +summary of the rule being proposed. + +If storage is passed to an `inout` argument, then any other simultaneous +attempt to read or write to that storage, including to the storage +containing it, will have have unspecified behavior. Reads from it may +see partially-updated values, or even values which will change as +modifications are made to the original storage; and writes may be +clobbered or simply disappear. + +But this only applies during the call with the `inout` argument: the +evaluation of other arguments to the call will not be interfered with, +and as soon as the call ends, all these modifications will resolve back +to a quiescent state. + +And this unspecified behavior has limits. The storage may end up with an +unexpected value, with only a subset of the writes made to it, and +copies from it may unexpectedly reflect modifications made after they +were copied. However, the program will otherwise remain in a consistent +and uncorrupted state. This means that execution will be able to +continue apace as long as these unexpected values don't trip up some +higher-level invariant. + +Tracking formal accesses +------------------------ + +Okay, now that I've analyzed this to death, it's time to make a concrete +proposal about the implementation. + +As discussed above, the safety hole with addressors can be fixed by +always retaining the buffer which keeps the address valid. Assuming that +other uses of the buffer follow the general copy-on-write pattern, this +retain will prevent structural changes to the buffer while the address +is in use. + +But, as I also discussed above, this introduces two problems: + +### Copies during modification + +Copying a COW aggregate value always shares the same buffer that was +stored there at the time of the copy; there is no uniqueness check done +as part of the copy. Changes to subobjects will then be instantly +reflected in the "copy" as they are made to the original. The structure +of the copy will stay the same, but the values of its subobjects will +appear to spontaneously change. + +I want to say that this behavior is acceptable according to the +formal-access rule I laid out above. How does that reasoning work? + +First, I need to establish what kind of behavior is at work here. It +clearly isn't guaranteed behavior: copies of COW values are normally +expected to be independent. The code wasn't rejected by the compiler, +nor did it dynamically assert; it simply seems to misbehave. But there +are limits to the misbehavior: + +- By general COW rules, there's no way to change the structure of an + existing buffer unless the retain count is 1. For the purposes of + this analysis, that means that, as long as the retain count is above + 1, there's no way to invalidate the address returned by + the addressor. +- The buffer will be retained for as long as the returned address is + being modified. This retain is independent of any storage which + might hold the aggregate value (and thus also retain the buffer). +- Because of this retain, the only way for the retain count to drop to + 1 is for no storage to continue to refer to the buffer. +- But if no storage refers to the buffer, there is no way to initiate + an operation which would change the buffer structure. + +Thus the address will remain valid, and there's no danger of memory +corruption. The only thing is that the program no longer makes useful +guarantees about the value of the copied aggregate. In other words, the +copy yielded an unspecified value. + +The formal-access rule allows loads from storage to yield an unspecified +value if there's another formal access to that storage in play and the +load is (1) not from an l-value derived from a name in the other FA's +DSN set and (2) not from a non-overlapping subobject. Are these +conditions true? + +Recall that an addressor is invoked for an l-value of the form: + + base.memory + +or: + + base[i] + +Both cases involve a formal access to the storage `base` as the base of +a subobject formal access. This kind of formal access always has an +empty DSN set, regardless of how the subobject is used. A COW mutable +addressor will always ensure that the buffer is uniquely referenced +before returning, so the only way that a value containing that buffer +can be copied is if the load is a non-subobject access to `base`. +Therefore, there are two simultaneous formal accesses to the same +storage, and the load is not from an l-value derived from the +modification's DSN set (which is empty), nor is it for a non-overlapping +subobject. So the formal-access rule applies, and an unspecified value +is an acceptable result. + +The implementation requirement here, then, is simply that the addressor +must be called, and the buffer retained, within the duration of the +formal access. In other words, the addressor must only be called +immediately prior to the call, rather than at the time of the formal +evaluation of the l-value expression. + +What would happen if there *were* a simultaneous load from a +non-overlapping subobject? Accessing the subobject might cause a brief +copy of `base`, but only for the duration of copying the subobject. If +the subobject does not overlap the subobject which was projected out for +the addressor, then this is harmless, because the addressor will not +allow modifications to those subobjects; there might be other +simultaneous formal accesses which do conflict, but these two do not. If +the subobject does overlap, then a recursive analysis must be applied; +but note that the exception to the formal-access rule will only apply if +non-overlapping subobjects were projected out from *both* formal +accesses. Otherwise, it will be acceptable for the access to the +overlapping subobject to yield an unspecified value. + +### Avoiding subobject clobbering during parallel modification + +The other problem is that the retain will prevent simultaneous changes +to the same buffer. The second change will cause a structural copy, and +the first address will end up modifying a buffer which is no longer +referenced: in other words, the program will observe subobject +clobbering. A similar analysis to the one from the last section suggests +that this can be described as unspecified behavior. + +Unfortunately, this unspecified behavior is unwanted: it violates the +guarantees of the formal-access rule as I laid it out above, because it +occurs even if you have formal accesses to two non-overlapping +subobjects. So something does need to be done here. + +One simple answer is to dynamically track whether a COW buffer is +currently undergoing a non-structural mutation. I'll call this *NSM +tracking*, and I'll call buffers which are undergoing non-structural +mutations *NSM-active*. + +The general rules of COW say that mutating operations must ensure that +their buffer is uniquely referenced before performing the modification. +NSM tracking works by having non-structural mutations perform a weaker +check: the buffer must be either uniquely referenced or be NSM-active. +If the non-structural mutation allows arbitrary code to run between the +start of the mutation and the end --- as an addressor does --- it must +both retain the buffer and flag it as NSM-active for the entire +duration. + +Because the retain still occurs, and because any *structural* changes to +the buffer that might invalidate the addresses of subobjects are still +blocked by that retain, all of the earlier analysis about the memory +safety of simultaneous accesses still applies. The only change is that +simultaneous non-structural modifications, as would be created by +simultaneous formal accesses to subobjects, will now be able to occur on +a single buffer. + +A set of simultaneous formal accesses on a single thread follows a +natural stack protocol, or can be made to do so with straightforward +SILGen and SIL optimizer consideration. Therefore, the runtime can track +whether a buffer is NSM-active on a thread using a single bit, which +nested modifications can be told not to clear. Call this the *NSM bit*. +Ignoring multithreading considerations for a moment, since the NSM bit +is only ever set at the same as a retain and only ever cleared at the +same time as a release, it makes sense to pack this into the strong +reference count. There is no need to support this operation on non-Swift +objects. The runtime should provide three new functions: + +- A function to test whether an object is either uniquely referenced + or NSM-active. Call this `swift_isUniquelyReferencedForNSM`. +- A function to perform the above test and, if the test passes and the + NSM bit is not set, atomically retain the object and set the + NSM bit. It should return both the result of the test and an object + to later set as NSM-inactive. That object will be nil if the test + failed or the NSM bit was already set. Call this + `swift_tryRetainForNSM`. +- A function to atomically clear the NSM bit and release the object. + Call this `swift_releaseForNSM`. + +These operations should also be reflected in SIL. + +#### Concurrent modifications and the non-structural modification bit + +What about concurrency? Two concurrent non-structural modifications +could race to set the NSM bit, and then the winning thread could clear +it before the other thread's modification is complete. This could cause +memory-unsafe behavior, since the losing thread would be modifying the +object through an address while not retaining the value. + +The major question here is whether this is a significant objection. It's +accepted that race conditions have undefined behavior. Is such code +inherently racy? + +The answer appears to be "no", and that it is possible to write code +which concurrently writes to existing non-overlapping elements of a COW +aggregate without causing races; but that such code is extremely +fraught, and moreover it is extremely fraught regardless of whether +NSM-activeness is tracked with a single bit or a wider count. Consider: + +- If the shared aggregate value is ever non-uniquely referenced, two + threads concurrently modifying it will race to unique the array. + This unavoidably has undefined behavior, because uniquing the array + requires the previous value to eventually be released, and a race + may cause an over-release. +- Assume that it's possible to guarantee that the aggregate value's + buffer is uniquely referenced before any threads concurrently + access it. Now, all of the threads are performing different + concurrent accesses. + - If any of the accesses is a structural modification, there will + be a race to re-unique the buffer. + - If all of the accesses are non-structural modifications, then + there will be no races as long as the retain-and-set and + release-and-clear operations are atomic: when starting any + particular operation, the buffer will always either be uniquely + referenced or have the bit set. + - If any of the accesses is a read, and that read does not occur + during a non-structural modification, then the buffer may + briefly become non-uniquely referenced and there will be a race + from concurrent modifications to re-unique it. + - If any of the accesses is a read, and that read occurs during a + non-structural modification, and the optimizer does not re-order + the read's retain/release around the retainForNSM/releaseForNSM + operations, then it matters how NSM-activeness is tracked. + + If there is complete tracking (i.e. a count, not just a single + bit), the retain for the read will only occur while the buffer + is flagged as NSM-active, and so it will have no effect. + + If there is incomplete tracking (i.e. just a single NSM bit), + then there is a potential for undefined behavior. Suppose two + threads race to set the NSM bit. The loser then initiates a read + and retains the buffer. Before the loser releases the buffer, + the winner clears the NSM bit. Now another thread might see that + the buffer is non-uniquely referenced and not NSM-active, and so + it will attempt to unique the buffer. + + It is probably unreasonable to require the optimizer to never + reorder ordinary retains and releases past retainForNSM and + releaseForNSM operations. + +More importantly, the use case here (many threads concurrently accessing +different elements of a shared data structure) just inherently doesn't +really work well with a COW data structure. Even if the library were +able to make enough guarantees to ensure that, with the right pattern of +accesses, there would never be a structural copy of the aggregate, it +would still be extremely inefficient, because all of the threads would +be competing for atomic access to the strong reference count. + +In short, I think it's reasonable for the library to say that programs +which want to do this should always use a type with reference semantics. +Therefore, it's reasonable to ignore concurrent accesses when deciding +how to best track whether an aggregate is undergoing non-structural +modification. This removes the only objection I can see to tracking this +with a single NSM bit. + +### Code generation patterns + +The signatures and access patterns for addressors will need to change in +order to ensure memory-safety. + +`mutableAddress` currently returns an `UnsafeMutablePointer`; it will +need to return `(Builtin.NativeObject?, UnsafeMutablePointer)`. The +owner pointer must be a native object; we cannot efficiently support +either uniqueness checking or the NSM bit on non-Swift objects. SILGen +will mark that the address depends on the owner reference and push a +cleanup to `releaseForNSM` it. + +`address` currently returns an `UnsafePointer`; it will need to return +`(Builtin.NativeObject?, UnsafePointer)`. I do not currently see a +reason to allow non-Swift owners, but the model doesn't depend on that. +SILGen will mark that the address depends on the owner reference and +push a cleanup to `release` it. + +In order to support ultimately calling an addressor in the conservative +access path, `materializeForSet` must also return an owner reference. +Since `materializeForSet` calls `mutableAddress` in this case, SILGen +will follow that pattern for calls. SILGen will also assume that the +need to perform a `releaseForNSM` is exclusive with the need to call the +setter. + +Mutating operations on COW types will now have two different paths for +making a buffer mutable and unique: one for structural mutations and +another for non-structural mutations. I expect that this will require +separate semantics annotations, and the optimizer will have to recognize +both. + +`releaseForNSM` operations will not be reorderable unless the optimizer +can prove that the objects are distinct. + +Summary of proposal and plan +---------------------------- + +Let me summarize what I'm proposing: + +- Swift's core approach to optimizing accesses should be based around + providing direct access to memory, either statically or dynamically. + In other words, Swift should adopt addressors on core data + structures as much as possible. +- Swift should fix the current memory hole with addressors by + retaining for the duration of the access and, for modifications, + flagging the buffer as NSM-active. The implementation plan follows: + - The runtime implements the NSM-bit and its entrypoints. + - SIL provides operations for manipulating and querying the + NSM bit. IRGen implements these operations using the + runtime functions. Builtins are exposed. + - The standard library changes data structures to do different + uniquing for structural and non-structural modifications. This + patch is not yet committed. + - The optimizer reacts to the above. When both are settled, they + can be committed. + - SILGen changes the emission patterns for l-values so that + addresses and writebacks are live only during the formal access. + - Sema changes the signature of `address`, `mutableAddress`, and + `materializeForSet` to return an optional owner reference. Sema + changes `materializeForSet` synthesis to return the + owner correctly. SILGen implements the desired code patterns. + + The standard library changes its addressor implementations to + continue to compile, but for staging purposes, it only uses + nil owners. + + - The standard library changes addressor implementations to use + meaningful owners. This patch is not yet committed. + - The optimizer reacts to the above. When both are settled, they + can be committed. diff --git a/docs/proposals/Accessors.rst b/docs/proposals/Accessors.rst deleted file mode 100644 index e4f0f19b82f93..0000000000000 --- a/docs/proposals/Accessors.rst +++ /dev/null @@ -1,1373 +0,0 @@ -:orphan: - -Optimizing Accessors in Swift -============================= - -Definitions ------------ - -An abstract storage declaration is a language construct that declares -a means of accessing some sort of abstract entity. I'll just say -"storage" hereafter. - -Swift provides three storage declarations: - -* a single named entity, called a *variable* and declared with ``var`` -* a single named entity which can never be reassigned, called a *constant* and declared with ``let`` -* a compound unnamed entity accessed with an index, called a *subscript* and declared with ``subscript`` - -These features are similar to those in other languages. Swift notably -lacks compound named entities, such as C#'s indexed properties; the -design team intentionally chose to favor the use of named single -entities of subscriptable type. - -It's useful to lump the two kinds of single named entities together; -I'll just call them both "variables" hereafter. - -Subscripts must always be instance type members. When a variable is -a type member, it's called a *property*. - -When I say that these entities are *abstract*, I mean that they're not -directly tied to any particular implementation. All of them may be -backed directly by memory storing values of the entity's type, or they -may simply provide a way of invoking arbitrary code to access a -logical memory, or they may be a little of both. - -Full-value accesses -------------------- - -All accesses to storage, no matter the implementation, can be performed -with two primitive operations: - -* a full-value load, which creates a new copy of the current value -* a full-value store, which overwrites the current value with a - different, independent value - -A function which implements a full-value load is called a *getter*; -a full-value store, a *setter*. - -An operation which calls for a full-value load into a temporary, then -a modification of the temporary, then a full-value store of the -temporary into the original entity, is called a *write-back*. - -Implementing accesses with full-value accesses introduces two -problems: subobject clobbering and performance. - -Subobject clobbering -~~~~~~~~~~~~~~~~~~~~ - -Subobject clobbering is a semantic issue. It occurs when there are -two changes to an entity in flight at once, as in:: - - swap(&point.x, &point.y) - -(By "at once", I mean synchronously. Unlike Java, Swift is willing to -say that an *asynchronous* simultaneous access (mutating or not) to an -entity that's being modified is completely undefined behavior, with -any sort of failure permitted, up to and including memory corruption -and crashes. As Swift transitions towards being a systems language, -it can selectively choose to define some of this behavior, -e.g. allowing racing accesses to different elements of an array.) - -Subobject clobbering is a problem with user expectation. Users are -unlikely to write code that intentionally modifies two obviously -overlapping entities, but they might very reasonably write code that -modifies two "separate" sub-entities. For example, they might write -code to swap two properties of a struct, or two elements of an array. -The natural expansion of these subobject accesses using whole-object -accesses generates code that "loses" the changes made to one of the -objects:: - - var point0 = point - var x = point0.x - var point1 = point - var y = point1.y - swap(&x, &y) - point1.y = y - point = point1 - point0.x = x - point = point0 - -Note that ``point.y`` is left unchanged. - -Local analysis -^^^^^^^^^^^^^^ - -There are two straightforward solutions to subobject clobbering, both -reliant on doing local analysis to recognize two accesses which -obviously alias (like the two references to ``point`` in the above -example). Once you've done this, you can either: - -* Emit an error, outlawing such code. This is what Swift currently - does (but only when the aliasing access must be implemented with - full-value loads and stores). -* Use a single temporary, potentially changing the semantics. - -It's impossible to guarantee the absence of subobject clobbering with -this analysis without extremely heavy-handed languages changes. -Fortunately, subobject clobbering is "only" a semantics issue, not a -memory-safety issue, at least as long as it's implemented with -full-value accesses. - -Reprojection -^^^^^^^^^^^^ - -A more general solution is to re-project the modified subobject from -scratch before writing it back. That is, you first acquire an initial -value like normal for the subobject you wish to modify, then modify -that temporary copy in-place. But instead of then recursively -consuming all the intermediate temporaries when writing them back, you -drop them all and recompute the current value, then write the modified -subobject to it, then write back all the way up. That is:: - - var point0 = point - var x = point0.x - var point1 = point - var y = point1.y - swap(&x, &y) - point1 = point // reload point1 - point1.y = y - point = point1 - point0 = point // reload point0 - point0.x = x - point = point0 - -In this example, I've applied the solution consistently to all -accesses, which protects against unseen modifications (e.g. during the -call to ``swap``) at the cost of performing two extra full-value -loads. - -You can heuristically lower this by combining it with a simple local -analysis and only re-projecting when writing back to other l-values -besides the last. In other words, generate code that will work as -long as the entity is not modified behind abstraction barriers or -through unexpected aliases:: - - var point0 = point - var x = point0.x - var point1 = point - var y = point1.y - swap(&x, &y) - point1.y = y // do not reload point1 - point = point1 - point0 = point // reload point0 - point0.x = x - point = point0 - -Note that, in either solution, you've introduced extra full-value -loads. This may be quite expensive, and it's not guaranteed to be -semantically equivalent. - -Performance -~~~~~~~~~~~ - -There are three major reasons why full-value accesses are inefficient. - -Unnecessary subobject accesses -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The first is that they may load or store more than is necessary. - -As an obvious example, imagine a variable of type ``(Int,Int)``; even -if my code only accesses the first element of the tuple, full-value -accesses force me to read or write the second element as well. That -means that, even if I'm purely overwriting the first element, I -actually have to perform a full-value load first so that I know what -value to use for the second element when performing the full-value -store. - -Additionally, while unnecessarily loading the second element of an -``(Int,Int)`` pair might seem trivial, consider that the tuple could -actually have twenty elements, or that the second element might be -non-trivial to copy (e.g. if it's a retainable pointer). - -Abstraction barriers -^^^^^^^^^^^^^^^^^^^^ - -A full-value load or store which you can completely reason about is one -thing, but if it has to be performed as a call, it can be a major -performance drag. - -For one, calls do carry a significant amount of low-level overhead. - -For another, optimizers must be extremely conservative about what a -call might do. A retainable pointer might have to be retained and -later released purely to protect against the possibility that a getter -might, somehow, cause the pointer to otherwise be deallocated. - -Furthermore, the conventions of the call might restrict performance. -One way or another, a getter for a retainable pointer generally -returns at +1, meaning that as part of the return, it is retained, -forcing the caller to later release. If the access were instead -direct to memory, this retain might be avoidable, depending on what -the caller does with the pointer. - -Copy-on-write -^^^^^^^^^^^^^ - -These problems are compounded by copy-on-write (COW) types. In Swift, -a copy-on-write value embeds an object reference. Copying the value -has low immediate cost, because it simply retains the existing -reference. However, modifying a value requires the reference to be -made unique, generally by copying the data held by the value into a -fresh object. I'll call this operation a *structural copy* in an -effort to avoid the more treacherous term "deep copy". - -COW types are problematic with full-value accesses for several reasons. - -First, COW types are often used to implement aggregates and thus often -have several distinguishable subobjects which users are likely to -think of as independent. This heightens the dangers of subobject -clobbering. - -Second, a full-value load of a COW type implies making the object -reference non-unique. Changing the value at this point will force a -structural copy. This means that modifying a temporary copy has -dramatically worse performance compared to modifying the original -entity in-place. For example:: - - window.name += " (closing)" - -If ``&window.name`` can be passed directly to the operator, and the -string buffer is uniquely referenced by that string, then this -operation may be as cheap as copying a few characters into the tail of -the buffer. But if this must be done with a write-back, then the -temporary will never have a unique reference, and there will always -be an unneeded structural copy. - -Conservative access patterns ----------------------------- - -When you know how storage is implemented, it's straightforward to -generate an optimal access to it. There are several major reasons why -you might not know how a storage declaration is implemented, though: - -* It might be an abstract declaration, not a concrete declaration. - Currently this means a protocol member, but Swift may someday add - abstract class members. - -* It might be a non-final class member, where the implementation you - can see is potentially overridable by a subclass. - -* It might be a resilient declaration, where you know only that the - entity exists and know nothing statically about its implementation. - -In all of these cases, you must generate code that will handle the -worst possible case, which is that the entity is implemented with a -getter and a setter. Therefore, the conservative access pattern -includes opaque getter and setter functions. - -However, for all the reasons discussed above, using unnecessary -full-value accesses can be terrible for performance. It's really bad -if a little conservatism --- e.g. because Swift failed to devirtualize -a property access --- causes asymptotic inefficiencies. Therefore, -Swift's native conservative access pattern also includes a third -accessor which permits direct access to storage when possible. This -accessor is called ``materializeForSet``. - -``materializeForSet`` receives an extra argument, which is an -uninitialized buffer of the value type, and it returns a pointer and a -flag. When it can provide direct access to storage for the entity, it -constructs a pointer to the storage and returns false. When it can't, -it performs a full-value load into the buffer and returns true. The -caller performs the modification in-place on the returned pointer and -then, if the flag is true, passes the value to the setter. - -The overall effect is to enable direct storage access as a dynamic -optimization when it's impossible as a static optimization. - -For now, ``materializeForSet`` is always automatically generated based -on whether the entity is implemented with a computed setter. It is -possible to imagine data structures that would benefit from having -this lifted to a user-definable feature; for example, a data structure -which sometimes holds its elements in memory but sometimes does not. - -``materializeForSet`` can provide direct access whenever an address -for the storage can be derived. This includes when the storage is -implemented with a ``mutableAddress`` accessor, as covered below. -Observing accessors currently prevent ``materializeForSet`` from -offering direct access; that's fixable for ``didSet`` using a slightly -different code pattern, but ``willSet`` is an inherent obstacle. - -Independent of any of the other optimizations discussed in this -whitepaper, ``materializeForSet`` had the potential to immediately -optimize the extremely important case of mutations to COW values in -un-devirtualized class properties, with fairly minimal risk. -Therefore, ``materializeForSet`` was implemented first, and it shipped -in Xcode 6.1. - -Direct access at computed addresses ------------------------------------ - -What entities can be directly accessed in memory? Non-computed -variables make up an extremely important set of cases; Swift has -enough built-in knowledge to know that it can provide direct access to -them. But there are a number of other important cases where the -address of an entity is not built-in to the compiler, but where direct -access is nonetheless possible. For example, elements of a simple -array always have independent storage in memory. Most benchmarks on -arrays would profit from being able to modify array elements in-place. - -There's a long chain of proposals in this area, many of which are -refinement on previous proposals. None of these proposals has yet -shipped in Xcode. - -Addressors -~~~~~~~~~~ - -For something like a simple array (or any similar structure, like a -deque) which is always backed by a buffer, it makes sense for the -implementor to simply define accessors which return the address of -the element. Such accessors are called *addressors*, and there are -two: ``address`` and ``mutableAddress``. - -The conservative access pattern can be generated very easily from -this: the getter calls ``address`` and loads from it, the setter calls -``mutableAddress`` and stores to it, and ``materializeForSet`` -provides direct access to the address returned from -``mutableAddress``. - -If the entity has type ``T``, then ``address`` returns an -``UnsafePointer`` and ``mutableAddress`` returns an -``UnsafeMutablePointer``. This means that the formal type of the -entity must exactly match the formal type of the storage. Thus, the -standard subscript on ``Dictionary`` cannot be implemented using -addressors, because the formal type of the entity is ``V?``, but the -backing storage holds a ``V``. (And this is in keeping with user -expectations about the data structure: assigning ``nil`` at a key is -supposed to erase any existing entry there, not create a new entry to -hold ``nil``.) - -This simple addressor proposal was the first prong of our efforts to -optimize array element access. Unfortunately, while it is useful for -several other types (such as ``ContiguousArray`` and -``UnsafeMutablePointer``), it is not flexible enough for the ``Array`` -type. - -Mixed addressors -~~~~~~~~~~~~~~~~ - -Swift's chief ``Array`` type is only a simple array when it is not -interacting with Objective-C. Type bridging requires ``Array`` to be -able to store an immutable ``NSArray`` instance, and the ``NSArray`` -interface does not expose the details of how it stores elements. An -``NSArray`` is even permitted to dynamically generate its values in -its ``objectAtIndex:`` method. And it would be absurd for ``Array`` -to perform a structural copy during a load just to make non-mutating -accesses more efficient! So the load access pattern for ``Array``'s -subscript declaration must use a getter. - -Fortunately, this requirement does not preclude using an addressor for -mutating accesses. Mutations to ``Array`` always transition the array -to a unique contiguous buffer representation as their first step. -This means that the subscript operator can sensibly return an address -when it's used for the purposes of mutation: in other words, exactly -when ``mutableAddress`` would be invoked. - -Therefore, the second prong of our efforts to optimize array element -access was to allow entities to be implemented with the combination of -a ``get`` accessor and a ``mutableAddress`` accessor. This is -straightforward in the user model, where it simply means lifting a -restriction. It's more complex behind the scenes because it broke -what was previously a clean conceptual division between "physical" and -"logical" l-values. - -Mixed addressors have now been adopted by ``Array`` to great success. -As expected, they substantially improved performance mutating COW -array elements. But they also fix an important instance of subobject -clobbering, because modifications to different subobjects (notably, -different elements of the same array) can occur simultaneously by -simply projecting out their addresses in the unique buffer. For -example, this means that it's possible to simply swap two elements -of an array directly:: - - swap(&array[i], &array[j]) - - // Expanded: - array.transitionToUniquelyReferenced() - let address_i = array.buffer.storage + i - array.transitionToUniquelyReferenced() - let address_j = array.buffer.storage + j - swap(address_i, address_j) - -Mixed addressors weren't completely implemented until very close to -the Xcode 6.1 deadline, and they changed code-generation patterns -enough to break a number of important array-specific optimizations. -Therefore, the team sensibly decided that they were too risky for that -release, and that there wasn't enough benefit from other applications -to justify including any of the addressor work. - -In a way, that was a fortunate decision, because the naive version of -addressors implemented so far in Swift creates a safety hole which -would otherwise have been exposed to users. - -Memory unsafety of addressors -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The semantics and memory safety of operations on COW types rely on a -pair of simple rules: - -* A non-mutating operation must own a reference to the buffer for - the full course of the read. - -* A mutating operation must own a unique reference to the buffer - for the full course of the mutation. - -Both rules tend to be naturally satisfied by the way that operations -are organized into methods. A value must own a reference to its -buffer at the moment that a method is invoked on it. A mutating -operation immediately transitions the buffer to a unique reference, -performing a structural copy if necessary. This reference will remain -valid for the rest of the method as long as the method is *atomic*: as -long as it does not synchronously invoke arbitrary user code. - -(This is a single-threaded notion of atomicity. A second thread which -modifies the value simultaneously can clearly invalidate the -assumption. But that would necessarily be a data race, and the -language design team is willing to say that such races have fully -undefined behavior, and arbitrary consequences like memory corruption -and crashes are acceptable in their wake.) - -However, addressors are not atomic in this way: they return an address -to the caller, which may then interleave arbitrary code before -completing the operation. This can present the opportunity for -corruption if the interleaved code modifies the original value. -Consider the following code:: - - func operate(inout value: Int, count: Int) { ... } - - var array: [Int] = [1,2,3,4] - operate(&array[0], { array = []; return 0 }()) - -The dynamic sequence of operations performed here will expand like so:: - - var array: [Int] = [1,2,3,4] - let address = array.subscript.mutableAddress(0) - array = [] - operate(address, 0) - -The assignment to ``array`` within the closure will release the buffer -containing ``address``, thus passing ``operate`` a dangling pointer. - -Nor can this be fixed with a purely local analysis; consider:: - - class C { var array: [Int] } - let global_C = C() - - func assign(inout value: Int) { - C.array = [] - value = 0 - } - - assign(&global_C.array[0]) - -Fixing the memory safety hole -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Conceptually, the correct fix is to guarantee that the rules are -satisfied by ensuring that the buffer is retained for the duration of -the operation. Any interleaving modifications will then see a -non-uniquely-referenced buffer and perform a structural copy:: - - // Project the array element. - let address = array.subscript.mutableAddress(0) - - // Remember the new buffer value and keep it retained. - let newArrayBuffer = array.buffer - retain(newArrayBuffer) - - // Reassign the variable. - release(array.buffer) - array.buffer = ... - - // Perform the mutation. These changes will be silently lost, but - // they at least won't be using deallocated memory. - operate(address, 0) - - // Release the "new" buffer. - release(newArrayBuffer) - -Note that this still leaves a semantic hole if the original value is -copied in interleaving code before the modification, because the -subsequent modification will be reflected in the copy:: - - // Project the array element. - let address = array.subscript.mutableAddress(0) - - // Remember the new buffer value and keep it retained. - let newArrayBuffer = array.buffer - retain(newArrayBuffer) - - // Copy the value. Note that arrayCopy uses the same buffer that - // 'address' points into. - let arrayCopy = array - retain(arrayCopy.buffer) - - // Perform the mutation. - operate(address, 0) - - // Release the "new" buffer. - release(newArrayBuffer) - -This might be unexpected behavior, but the language team is willing to -accept unexpected behavior for this code. What's non-negotiable is -breaking memory safety. - -Unfortunately, applying this fix naively reintroduces the problem of -subobject clobbering: since a modification of one subobject -immediately retains a buffer that's global to the entire value, an -interleaved modification of a different subobject will see a -non-unique buffer reference and therefore perform a structural copy. -The modifications to the first subobject will therefore be silently -lost. - -Unlike the interleaving copy case, this is seen as unacceptable. -Notably, it breaks swapping two array elements:: - - // Original: - swap(&array[i], &array[j]) - - // Expanded: - - // Project array[i]. - array.transitionToUniquelyReferenced() - let address_i = array.buffer.storage + i - let newArrayBuffer_i = array.buffer - retain(newArrayBuffer_i) - - // Project array[j]. Note that this transition is guaranteed - // to have to do a structural copy. - array.transitionToUniquelyReferenced() - let address_j = array.buffer.storage + j - let newArrayBuffer_j = array.buffer - retain(newArrayBuffer_j) - - // Perform the mutations. - swap(address_i, address_j) - - // Balance out the retains. - release(newArrayBuffer_j) - release(newArrayBuffer_i) - -Acceptability -------------- - -This whitepaper has mentioned several times that the language team is -prepared to accept such-and-such behavior but not prepared to accept -some other kind of behavior. Clearly, there is a policy at work. -What is it? - -General philosophy -~~~~~~~~~~~~~~~~~~ - -For any given language problem, a perfect solution would be one which: - -* guarantees that all operations complete without crashing or - corrupting the program state, - -* guarantees that all operations produce results according to - consistent, reliable, and intuitive rules, - -* does not limit or require complex interactions with the remainder - of the language, and - -* imposes no performance cost. - -These goals are, however, not all simultaneously achievable, and -different languages reach different balances. Swift's particular -philosophy is as follows: - -* The language should be as dynamically safe as possible. - Straightforward uses of ordinary language features may cause - dynamic failure, but the should never corrupt the program state. - Any unsafe language or library features (other than simply calling - into C code) should be explicitly labeled as unsafe. - - A dynamic failure should mean that the program reliably halts, - ideally with a message clearly describing the source of the - failure. In the future, the language may allow for emergency - recovery from such failures. - -* The language should sit on top of C, relying only on a relatively - unobtrusive runtime. Accordingly, the language's interactions - with C-based technologies should be efficient and obvious. - -* The language should allow a static compiler to produce efficient - code without dynamic instrumentation. Accordingly, static - analysis should only be blocked by incomplete information when - the code uses an obviously abstract language feature (such as - calling a class method or an unknown function), and the language - should provide tools to allow programmers to limit such cases. - - (Dynamic instrumentation can, of course, still help, but it - shouldn't be required for excellent performance.) - -General solutions -~~~~~~~~~~~~~~~~~ - -A language generally has six tools for dealing with code it considers -undesireable. Some of this terminology is taken from existing -standards, others not. - -* The language may nonetheless take steps to ensure that the code - executes with a reliable result. Such code is said to have - *guaranteed behavior*. - -* The language may report the code as erroneous before it executes. - Such code is said to be *ill formed*. - -* The language may reliably report the code as having performed an - illegal operation when it executes. Such code is said to be - *asserting* or *aborting*. - -* The language may allow the code to produce an arbitrary-but-sound - result. Such code is said to have *unspecified behavior* or to - have produced an *unspecified value*. - -* The language may allow the code to produce an unsound result which - will result in another of these behaviors, but only if used. - Such code is said to have produced a *trap value*. - -* The language may declare the code to be completely outside of the - guarantees of the language. Such code is said to have - *undefined behavior*. - -In keeping with its design philosophy, Swift has generally limited -itself to the first four solutions, with two significant exceptions. - -The first exception is that Swift provides several explicitly unsafe -language and library features, such as ``UnsafePointer`` and -``unowned(unsafe)``. The use of these features is generally subject -to undefined behavior rules. - -The second exception is that Swift does not make any guarantees about -programs in the presence of race conditions. It is extremely -difficult to make even weak statements about the behavior of a program -with a race condition without either: - -* heavily restricting shared mutable state on a language level, which - would require invasive changes to how the language interacts with C; - -* forcing implicit synchronization when making any change to - potentially shared memory, which would cripple performance and - greatly complicate library implementation; or - -* using a garbage collector to manage all accessible memory, which - would impose a very large burden on almost all of Swift's language - goals. - -Therefore, Swift does surrender safety in the presence of races. - -Acceptability conditions for storage accesses -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Storage access involves a tension between four goals: - -* Preserving all changes when making simultaneous modifications to - distinct subobjects; in other words, avoiding subobject clobbering - -* Performing a predictable and intuitive sequence of operations when - modifying storage that's implemented with a getter and setter - -* Avoiding unnecessary copies of a value during a modification, - especially when this forces a structural copy of a COW value - -* Avoiding memory safety holes when accessing storage that's been - implemented with memory. - -Reprojection_ is good at preserving changes, but it introduces extra -copies, and it's less intuitive about how many times getters and -setters will be called. There may be a place for it anyway, if we're -willing to accept the extra conceptual complexity for computed -storage, but it's not a reasonable primary basis for optimizing the -performance of storage backed by memory. - -Solutions permitting in-place modification are more efficient, but -they do have the inherent disadvantage of having to vend the address -of a value before arbitrary interleaving code. Even if the address -remains valid, and the solution to that avoids subobject clobbering, -there's an unavoidable issue that the write can be lost because the -address became dissociated from the storage. For example, if your -code passes ``&array[i]`` to a function, you might plausibly argue -that changes to that argument should show up in the ``i``\ th element of -``array`` even if you completely reassign ``array``. Reprojection_ -could make this work, but in-place solutions cannot efficiently do so. -So, for any in-place solution to be acceptable, there does need to be -some rule specifying when it's okay to "lose track" of a change. - -Furthermore, the basic behavior of COW means that it's possible to -copy an array with an element under modification and end up sharing -the same buffer, so that the modification will be reflected in a value -that was technically copied beforehand. For example:: - - var array = [1,2,3] - var oldArray : [Int] = [] - - // This function copies array before modifying it, but because that - // copy is of an value undergoing modification, the copy will use - // the same buffer and therefore observe updates to the element. - func foo(inout element: Int) { - oldArray = array - element = 4 - } - - // Therefore, oldArray[2] will be 4 after this call. - foo(&array[2]) - -Nor can this be fixed by temporarily moving the modified array aside, -because that would prevent simultaneous modifications to different -elements (and, in fact, likely cause them to assert). So the rule -will also have to allow this. - -However, both of these possibilities already come up in the design of -both the library and the optimizer. The optimizer makes a number of -assumptions about aliasing; for example, the general rule is that -storage bound to an ``inout`` parameter cannot be accessed through -other paths, and while the optimizer is not permitted to compromise -memory safety, it is permitted to introduce exactly this kind of -unexpected behavior where aliasing accesses may or may not the storage -as a consistent entity. - -Formal accesses -^^^^^^^^^^^^^^^ - -That rule leads to an interesting generalization. Every modification -of storage occurs during a *formal access* (FA) to that storage. An -FA is also associated with zero or more *designated storage names* -(DSNs), which are ``inout`` arguments in particular execution records. -An FA arises from an l-value expression, and its duration and DSN set -depend on how the l-value is used: - -* An l-value which is simply loaded from creates an instantaneous FA - at the time of the load. The DSN set is empty. - - Example:: - - foo(array) - // instantaneous FA reading array - -* An l-value which is assigned to with ``=`` creates an - instantaneous FA at the time of the primitive assignment. The DSN - set is empty. - - Example:: - - array = [] - // instantaneous FA assigning array - - Note that the primitive assignment strictly follows the evaluation - of both the l-value and r-value expressions of the assignment. - For example, the following code:: - - // object is a variable of class type - object.array = object.array + [1,2,3] - - produces this sequence of formal accesses:: - - // instantaneous FA reading object (in the left-hand side) - // instantaneous FA reading object (in the right-hand side) - // instantaneous FA reading object.array (in the right-hand side) - // evaluation of [1,2,3] - // evaluation of + - // instantaneous FA assigning object.array - -* An l-value which is passed as an ``inout`` argument to a call - creates an FA beginning immediately before the call and ending - immediately after the call. (This includes calls where an - argument is implicitly passed ``inout``, such as to mutating - methods or user-defined assignment operators such as ``+=`` or - ``++``.) The DSN set contains the ``inout`` argument within the - call. - - Example:: - - func swap(inout lhs: T, inout rhs: T) {} - - // object is a variable of class type - swap(&leftObject.array, &rightObject.array) - - This results in the following sequence of formal accesses:: - - // instantaneous FA reading leftObject - // instantaneous FA reading rightObject - // begin FA for inout argument leftObject.array (DSN={lhs}) - // begin FA for inout argument rightObject.array (DSN={rhs}) - // evaluation of swap - // end FA for inout argument rightObject.array - // end FA for inout argument leftObject.array - -* An l-value which is used as the base of a member storage access - begins an FA whose duration is the same as the duration of the FA - for the subobject l-value. The DSN set is empty. - - Example:: - - swap(&leftObject.array[i], &rightObject.array[j]) - - This results in the following sequence of formal accesses:: - - // instantaneous FA reading leftObject - // instantaneous FA reading i - // instantaneous FA reading rightObject - // instantaneous FA reading j - // begin FA for subobject base leftObject.array (DSN={}) - // begin FA for inout argument leftObject.array[i] (DSN={lhs}) - // begin FA for subobject base rightObject.array (DSN={}) - // begin FA for inout argument rightObject.array[j] (DSN={rhs}) - // evaluation of swap - // end FA for subobject base rightObject.array[j] - // end FA for inout argument rightObject.array - // end FA for subobject base leftObject.array[i] - // end FA for inout argument leftObject.array - -* An l-value which is the base of an ! operator begins an FA whose - duration is the same the duration of the FA for the resulting - l-value. The DSN set is empty. - - Example:: - - // left is a variable of type T - // right is a variable of type T? - swap(&left, &right!) - - This results in the following sequence of formal accesses:: - - // begin FA for inout argument left (DSN={lhs}) - // begin FA for ! operand right (DSN={}) - // begin FA for inout argument right! (DSN={rhs}) - // evaluation of swap - // end FA for inout argument right! - // end FA for ! operand right - // end FA for inout argument left - -* An l-value which is the base of a ? operator begins an FA whose - duration begins during the formal evaluation of the l-value - and ends either immediately (if the operand was nil) or at the - end of the duration of the FA for the resulting l-value. - In either case, the DSN set is empty. - - Example:: - - // left is a variable of optional struct type - // right is a variable of type Int - left?.member += right - - This results in the following sequence of formal accesses, assuming - that ``left`` contains a value:: - - // begin FA for ? operand left (DSN={}) - // instataneous FA reading right (DSN={}) - // begin FA for inout argument left?.member (DSN={lhs}) - // evaluation of += - // end FA for inout argument left?.member - // end FA for ? operand left - -The FAs for all ``inout`` arguments to a call begin simultaneously at -a point strictly following the evaluation of all the argument -expressions. For example, in the call ``foo(&array, array)``, the -evaluation of the second argument produces a defined value, because -the FA for the first argument does not begin until after all the -arguments are formally evaluated. No code should actually be emitted -during the formal evaluation of ``&array``, but for an expression like -``someClassReference.someArray[i]``, the class r-value and index -expressions would be fully evaluated at that time, and then the -l-value would be kept abstract until the FA begins. Note that this -requires changes in SILGen's current code generation patterns. - -The FA rule for the chaining operator ``?`` is an exception to the -otherwise-simple intuition that formal accesses begin immediately -before the modification begins. This is necessary because the -evaluation rules for ``?`` may cause arbitrary computation to be -short-circuited, and therefore the operand must be accessed during the -formal evaluation of the l-value. There were three options here: - -* Abandon short-circuiting for assignments to optional l-values. This - is a very high price; short-circuiting fits into user intuitions - about the behavior of the chaining operator, and it can actually be - quite awkward to replicate with explicit accesses. - -* Short-circuit using an instantaneous formal access, then start a - separate formal access before the actual modification. In other - words, evaluation of ``X += Y`` would proceed by first determining - whether ``X`` exists (capturing the results of any r-value - components), discarding any projection information derived from - that, evaluating ``Y``, reprojecting ``X`` again (using the saved - r-value components and checking again for whether the l-value - exists), and finally calling the ``+=`` operator. - - If ``X`` involves any sort of computed storage, the steps required - to evaluate this might be... counter-intuitive. - -* Allow the formal access to begin during the formal evaluation of the - l-value. This means that code like the following will have - unspecified behavior:: - - array[i]?.member = deriveNewValueFrom(array) - -In the end, I've gone with the third option. The intuitive -explanation is that ``array`` has to be accessed early in order to -continue the evaluation of the l-value. I think that's -comprehensible to users, even if it's not immediately obvious. - -Disjoint and non-disjoint formal accesses -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -I'm almost ready to state the core rule about formal accesses, but -first I need to build up a few more definitions. - -An *abstract storage location* (ASL) is: - -* a global variable declaration; - -* an ``inout`` parameter declaration, along with a reference - to a specific execution record for that function; - -* a local variable declaration, along with a reference to a - specific execution record for that declaration statement; - -* a static/class property declaration, along with a type having - that property; - -* a struct/enum instance property declaration, along with an - ASL for the base; - -* a struct/enum subscript declaration, along with a concrete index - value and an ASL for the base; - -* a class instance property declaration, along with an instance of - that class; or - -* a class instance subscript declaration, along with a concrete - index value and an instance of that class. - -Two abstract storage locations may be said to *overlap*. Overlap -corresponds to the imprecise intuition that a modification of one -location directly alters the value of another location. Overlap -is an "open" property of the language: every new declaration may -introduce its own overlap behavior. However, the language and -library make certain assertions about the overlap of some locations: - -* An ``inout`` parameter declaration overlaps exactly the set - of ASLs overlapped by the ASL which was passed as an argument. - -* If two ASLs are both implemented with memory, then they overlap - only if they have the same kind in the above list and the - corresponding data match: - - * execution records must represent the same execution - * types must be the same - * class instances must be the same - * ASLs must overlap - -* For the purposes of the above rule, the subscript of a standard - library array type is implemented with memory, and the two - indexes match if they have the same integer value. - -* For the purposes of the above rule, the subscript of a standard - library dictionary type is implemented with memory, and the two - indexes match if they compare equal with ``==``. - -Because this definition is open, it is impossible to completely -statically or dynamically decided it. However, it would still be -possible to write a dynamic analysis which decided it for common -location kinds. Such a tool would be useful as part of, say, an -ASan-like dynamic tool to diagnose violations of the -unspecified-behavior rule below. - -The overlap rule is vague about computed storage partly because -computed storage can have non-obvious aliasing behavior and partly -because the subobject clobbering caused by the full-value accesses -required by computed storage can introduce unexpected results that can -be reasonably glossed as unspecified values. - -This notion of abstract storage location overlap can be applied to -formal accesses as well. Two FAs ``x`` and ``y`` are said to be -*disjoint* if: - -* they refer to non-overlapping abstract storage locations or - -* they are the base FAs of two disjoint member storage accesses - ``x.a`` and ``y.b``. - -Given these definitions, the core unspecified-behavior rule is: - - If two non-disjoint FAs have intersecting durations, and neither FA - is derived from a DSN for the other, then the program has - unspecified behavior in the following way: if the second FA is a - load, it yields an unspecified value; otherwise, both FAs store an - unspecified value in the storage. - -Note that you cannot have two loads with intersecting durations, -because the FAs for loads are instantaneous. - -Non-overlapping subobject accesses make the base accesses disjoint -because otherwise code like ``swap(&a[0], &a[1])`` would have -unspecified behavior, because the two base FAs are to clearly -overlapping locations and have intersecting durations. - -Note that the optimizer's aliasing rule falls out from this rule. If -storage has been bound as an ``inout`` argument, accesses to it -through any path not derived from the ``inout`` argument will start a -new FA for overlapping storage, the duration of which will necessarily -intersect duration with that of the FA through which the ``inout`` -argument was bound, causing unspecified behavior. If the ``inout`` -argument is forwarded to another call, that will start a new FA which -is validly based on a DSN of the first; but an attempt to modify the -storage through the first ``inout`` argument while the second call is -active will create a third FA not based on the DSN from the second -``inout`` call, causing a conflict there. Therefore a function may -assume that it can see all accesses to the storage bound to an -``inout`` argument. - -If you didn't catch all that... -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -That may have been a somewhat intense description, so here's a simple -summary of the rule being proposed. - -If storage is passed to an ``inout`` argument, then any other -simultaneous attempt to read or write to that storage, including to -the storage containing it, will have have unspecified behavior. Reads -from it may see partially-updated values, or even values which will -change as modifications are made to the original storage; and writes -may be clobbered or simply disappear. - -But this only applies during the call with the ``inout`` argument: the -evaluation of other arguments to the call will not be interfered with, -and as soon as the call ends, all these modifications will resolve -back to a quiescent state. - -And this unspecified behavior has limits. The storage may end up with -an unexpected value, with only a subset of the writes made to it, and -copies from it may unexpectedly reflect modifications made after they -were copied. However, the program will otherwise remain in a -consistent and uncorrupted state. This means that execution will be -able to continue apace as long as these unexpected values don't trip -up some higher-level invariant. - -Tracking formal accesses ------------------------- - -Okay, now that I've analyzed this to death, it's time to make a -concrete proposal about the implementation. - -As discussed above, the safety hole with addressors can be fixed by -always retaining the buffer which keeps the address valid. Assuming -that other uses of the buffer follow the general copy-on-write -pattern, this retain will prevent structural changes to the buffer -while the address is in use. - -But, as I also discussed above, this introduces two problems: - -Copies during modification -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Copying a COW aggregate value always shares the same buffer that was -stored there at the time of the copy; there is no uniqueness check -done as part of the copy. Changes to subobjects will then be -instantly reflected in the "copy" as they are made to the original. -The structure of the copy will stay the same, but the values of -its subobjects will appear to spontaneously change. - -I want to say that this behavior is acceptable according to the -formal-access rule I laid out above. How does that reasoning work? - -First, I need to establish what kind of behavior is at work here. It -clearly isn't guaranteed behavior: copies of COW values are normally -expected to be independent. The code wasn't rejected by the compiler, -nor did it dynamically assert; it simply seems to misbehave. But -there are limits to the misbehavior: - -* By general COW rules, there's no way to change the structure of an - existing buffer unless the retain count is 1. For the purposes of - this analysis, that means that, as long as the retain count is - above 1, there's no way to invalidate the address returned by the - addressor. - -* The buffer will be retained for as long as the returned address - is being modified. This retain is independent of any storage - which might hold the aggregate value (and thus also retain the buffer). - -* Because of this retain, the only way for the retain count to drop - to 1 is for no storage to continue to refer to the buffer. - -* But if no storage refers to the buffer, there is no way to - initiate an operation which would change the buffer structure. - -Thus the address will remain valid, and there's no danger of memory -corruption. The only thing is that the program no longer makes useful -guarantees about the value of the copied aggregate. In other words, -the copy yielded an unspecified value. - -The formal-access rule allows loads from storage to yield an -unspecified value if there's another formal access to that storage in -play and the load is (1) not from an l-value derived from a name in -the other FA's DSN set and (2) not from a non-overlapping subobject. -Are these conditions true? - -Recall that an addressor is invoked for an l-value of the form:: - - base.memory - -or:: - - base[i] - -Both cases involve a formal access to the storage ``base`` as the base -of a subobject formal access. This kind of formal access always has -an empty DSN set, regardless of how the subobject is used. A COW -mutable addressor will always ensure that the buffer is uniquely -referenced before returning, so the only way that a value containing -that buffer can be copied is if the load is a non-subobject access to -``base``. Therefore, there are two simultaneous formal accesses to -the same storage, and the load is not from an l-value derived from the -modification's DSN set (which is empty), nor is it for a -non-overlapping subobject. So the formal-access rule applies, and -an unspecified value is an acceptable result. - -The implementation requirement here, then, is simply that the -addressor must be called, and the buffer retained, within the duration -of the formal access. In other words, the addressor must only -be called immediately prior to the call, rather than at the time -of the formal evaluation of the l-value expression. - -What would happen if there *were* a simultaneous load from a -non-overlapping subobject? Accessing the subobject might cause a -brief copy of ``base``, but only for the duration of copying the -subobject. If the subobject does not overlap the subobject which was -projected out for the addressor, then this is harmless, because the -addressor will not allow modifications to those subobjects; there -might be other simultaneous formal accesses which do conflict, but -these two do not. If the subobject does overlap, then a recursive -analysis must be applied; but note that the exception to the -formal-access rule will only apply if non-overlapping subobjects were -projected out from *both* formal accesses. Otherwise, it will be -acceptable for the access to the overlapping subobject to yield an -unspecified value. - -Avoiding subobject clobbering during parallel modification -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The other problem is that the retain will prevent simultaneous changes -to the same buffer. The second change will cause a structural copy, -and the first address will end up modifying a buffer which is no -longer referenced: in other words, the program will observe subobject -clobbering. A similar analysis to the one from the last section -suggests that this can be described as unspecified behavior. - -Unfortunately, this unspecified behavior is unwanted: it violates the -guarantees of the formal-access rule as I laid it out above, because -it occurs even if you have formal accesses to two non-overlapping -subobjects. So something does need to be done here. - -One simple answer is to dynamically track whether a COW buffer is -currently undergoing a non-structural mutation. I'll call this *NSM -tracking*, and I'll call buffers which are undergoing non-structural -mutations *NSM-active*. - -The general rules of COW say that mutating operations must ensure that -their buffer is uniquely referenced before performing the -modification. NSM tracking works by having non-structural mutations -perform a weaker check: the buffer must be either uniquely referenced -or be NSM-active. If the non-structural mutation allows arbitrary -code to run between the start of the mutation and the end --- as an -addressor does --- it must both retain the buffer and flag it as -NSM-active for the entire duration. - -Because the retain still occurs, and because any *structural* changes -to the buffer that might invalidate the addresses of subobjects are -still blocked by that retain, all of the earlier analysis about the -memory safety of simultaneous accesses still applies. The only change -is that simultaneous non-structural modifications, as would be created -by simultaneous formal accesses to subobjects, will now be able to -occur on a single buffer. - -A set of simultaneous formal accesses on a single thread follows a -natural stack protocol, or can be made to do so with straightforward -SILGen and SIL optimizer consideration. Therefore, the runtime can -track whether a buffer is NSM-active on a thread using a single bit, -which nested modifications can be told not to clear. Call this the -*NSM bit*. Ignoring multithreading considerations for a moment, since -the NSM bit is only ever set at the same as a retain and only ever -cleared at the same time as a release, it makes sense to pack this -into the strong reference count. There is no need to support this -operation on non-Swift objects. The runtime should provide three new -functions: - -* A function to test whether an object is either uniquely referenced - or NSM-active. Call this ``swift_isUniquelyReferencedForNSM``. - -* A function to perform the above test and, if the test passes and - the NSM bit is not set, atomically retain the object and set - the NSM bit. It should return both the result of the test and an - object to later set as NSM-inactive. That object will be nil if - the test failed or the NSM bit was already set. Call this - ``swift_tryRetainForNSM``. - -* A function to atomically clear the NSM bit and release the object. - Call this ``swift_releaseForNSM``. - -These operations should also be reflected in SIL. - -Concurrent modifications and the non-structural modification bit -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -What about concurrency? Two concurrent non-structural modifications -could race to set the NSM bit, and then the winning thread could clear -it before the other thread's modification is complete. This could -cause memory-unsafe behavior, since the losing thread would be -modifying the object through an address while not retaining the value. - -The major question here is whether this is a significant objection. -It's accepted that race conditions have undefined behavior. Is such -code inherently racy? - -The answer appears to be "no", and that it is possible to write code -which concurrently writes to existing non-overlapping elements of a -COW aggregate without causing races; but that such code is extremely -fraught, and moreover it is extremely fraught regardless of whether -NSM-activeness is tracked with a single bit or a wider count. Consider: - -* If the shared aggregate value is ever non-uniquely referenced, two - threads concurrently modifying it will race to unique the array. - This unavoidably has undefined behavior, because uniquing the - array requires the previous value to eventually be released, and a - race may cause an over-release. - -* Assume that it's possible to guarantee that the aggregate value's - buffer is uniquely referenced before any threads concurrently - access it. Now, all of the threads are performing different - concurrent accesses. - - * If any of the accesses is a structural modification, there will - be a race to re-unique the buffer. - - * If all of the accesses are non-structural modifications, then - there will be no races as long as the retain-and-set and - release-and-clear operations are atomic: when starting any - particular operation, the buffer will always either be uniquely - referenced or have the bit set. - - * If any of the accesses is a read, and that read does not occur - during a non-structural modification, then the buffer may - briefly become non-uniquely referenced and there will be a - race from concurrent modifications to re-unique it. - - * If any of the accesses is a read, and that read occurs during a - non-structural modification, and the optimizer does not re-order - the read's retain/release around the retainForNSM/releaseForNSM - operations, then it matters how NSM-activeness is tracked. - - If there is complete tracking (i.e. a count, not just a single - bit), the retain for the read will only occur while the buffer - is flagged as NSM-active, and so it will have no effect. - - If there is incomplete tracking (i.e. just a single NSM bit), - then there is a potential for undefined behavior. Suppose two - threads race to set the NSM bit. The loser then initiates a - read and retains the buffer. Before the loser releases the - buffer, the winner clears the NSM bit. Now another thread might - see that the buffer is non-uniquely referenced and not - NSM-active, and so it will attempt to unique the buffer. - - It is probably unreasonable to require the optimizer to never - reorder ordinary retains and releases past retainForNSM and - releaseForNSM operations. - -More importantly, the use case here (many threads concurrently -accessing different elements of a shared data structure) just -inherently doesn't really work well with a COW data structure. Even -if the library were able to make enough guarantees to ensure that, -with the right pattern of accesses, there would never be a structural -copy of the aggregate, it would still be extremely inefficient, -because all of the threads would be competing for atomic access to the -strong reference count. - -In short, I think it's reasonable for the library to say that programs -which want to do this should always use a type with reference -semantics. Therefore, it's reasonable to ignore concurrent accesses -when deciding how to best track whether an aggregate is undergoing -non-structural modification. This removes the only objection I -can see to tracking this with a single NSM bit. - -Code generation patterns -~~~~~~~~~~~~~~~~~~~~~~~~ - -The signatures and access patterns for addressors will need to change -in order to ensure memory-safety. - -``mutableAddress`` currently returns an ``UnsafeMutablePointer``; it -will need to return ``(Builtin.NativeObject?, UnsafeMutablePointer)``. -The owner pointer must be a native object; we cannot efficiently -support either uniqueness checking or the NSM bit on non-Swift -objects. SILGen will mark that the address depends on the owner -reference and push a cleanup to ``releaseForNSM`` it. - -``address`` currently returns an ``UnsafePointer``; it will need to -return ``(Builtin.NativeObject?, UnsafePointer)``. I do not currently -see a reason to allow non-Swift owners, but the model doesn't depend -on that. SILGen will mark that the address depends on the owner -reference and push a cleanup to ``release`` it. - -In order to support ultimately calling an addressor in the -conservative access path, ``materializeForSet`` must also return an -owner reference. Since ``materializeForSet`` calls ``mutableAddress`` -in this case, SILGen will follow that pattern for calls. SILGen will -also assume that the need to perform a ``releaseForNSM`` is exclusive -with the need to call the setter. - -Mutating operations on COW types will now have two different paths for -making a buffer mutable and unique: one for structural mutations and -another for non-structural mutations. I expect that this will require -separate semantics annotations, and the optimizer will have to -recognize both. - -``releaseForNSM`` operations will not be reorderable unless the -optimizer can prove that the objects are distinct. - -Summary of proposal and plan ----------------------------- - -Let me summarize what I'm proposing: - -* Swift's core approach to optimizing accesses should be based - around providing direct access to memory, either statically or - dynamically. In other words, Swift should adopt addressors on - core data structures as much as possible. - -* Swift should fix the current memory hole with addressors by - retaining for the duration of the access and, for modifications, - flagging the buffer as NSM-active. The implementation plan - follows: - - * The runtime implements the NSM-bit and its entrypoints. - - * SIL provides operations for manipulating and querying the NSM - bit. IRGen implements these operations using the runtime - functions. Builtins are exposed. - - * The standard library changes data structures to do different - uniquing for structural and non-structural modifications. This - patch is not yet committed. - - * The optimizer reacts to the above. When both are settled, they - can be committed. - - * SILGen changes the emission patterns for l-values so that - addresses and writebacks are live only during the formal - access. - - * Sema changes the signature of ``address``, ``mutableAddress``, - and ``materializeForSet`` to return an optional owner reference. - Sema changes ``materializeForSet`` synthesis to return the - owner correctly. SILGen implements the desired code patterns. - - The standard library changes its addressor implementations - to continue to compile, but for staging purposes, it only uses - nil owners. - - * The standard library changes addressor implementations to - use meaningful owners. This patch is not yet committed. - - * The optimizer reacts to the above. When both are settled, they - can be committed. diff --git a/docs/proposals/ArrayBridge.md b/docs/proposals/ArrayBridge.md new file mode 100644 index 0000000000000..6ba5cac8a1704 --- /dev/null +++ b/docs/proposals/ArrayBridge.md @@ -0,0 +1,123 @@ +Bridging Swift Arrays to/from Cocoa +=================================== + +Authors + +: Chris Lattner, Joe Groff, Dave Abrahams + +Summary + +: Unifying a fast C-style array with a Cocoa class cluster that can + represent arbitrarily complex data structures is challenging. In a + space where no approach satisfies all desires, we believe we've + found a good compromise. + +Basic Requirements +------------------ + +A successfully-bridged array type would be both "great for Cocoa" and +"great for C." + +Being "great for Cocoa" means this must work and be efficient: + + var a = [cocoaObject1, cocoaObject2] + someCocoaObject.takesAnNSArray(a) + + func processViews(views: AnyObject[]) { ... } + var b = someNSWindow.views // views is an NSArray + processViews(b) + + var c: AnyObject[] = someNSWindow.views + +Being "great For C" means that an array created in Swift must have +C-like performance and be representable as a base pointer and length, +for interaction with C APIs, at zero cost. + +Proposed Solution +----------------- + +`Array`, a.k.a. `T[]`, is notionally an `enum` with two cases; call +them `Native` and `Cocoa`. The `Native` case stores a `ContiguousArray`, +which has a known, contiguous buffer representation and O(1) access to +the address of any element. The `Cocoa` case stores an `NSArray`. + +`NSArray` bridges bidirectionally in O(1) [^1] to `AnyObject[]`. It also +implicitly converts in to `T[]`, where T is any class declared to be +`@objc`. No dynamic check of element types is ever performed for arrays +of `@objc` elements; instead we simply let `objc_msgSend` fail when +`T`'s API turns out to be unsupported by the object. Any `T[]`, where T +is an `@objc` class, converts implicitly to NSArray. + +Optimization +------------ + +Any type with more than one representation naturally penalizes +fine-grained operations such as indexing, because the cost of repeatedly +branching to handle each representation becomes significant. For +example, the design above would pose significant performance problems +for arrays of integers, because every subscript operation would have to +check to see if the representation is an NSArray, realize it is not, +then do the constant time index into the native representation. Beyond +requiring an extra check, this check would disable optimizations that +can provide a significant performance win (like auto-vectorization). + +However, the inherent limitations of `NSArray` mean that we can often +know at compile-time which representation is in play. So the plan is to +teach the compiler to optimize for the `Native` case unless the element +type is an `@objc` class or AnyObject. When `T` is statically known not +to be an `@objc` class or AnyObject, it will be possible to eliminate +the `Cocoa` case entirely. When generating code for generic algorithms, +we can favor the `Native` case, perhaps going so far as to specialize +for the case where all parameters are non-`@objc` classes. This will +give us C-like performance for array operations on `Int`, `Float`, and +other `struct` types [^2]. + +To implement this, we'll need to implement a new generic builtin, +something along the lines of "`Builtin.couldBeObjCType()`", which +returns a `Builtin.Int1` value. SILCombine and IRGen should eagerly fold +this to "0" iff `T` is known to be a protocol other than AnyObject, if +it is known to be a non-`@objc` class, or if it is known to be any +struct, enum or tuple. Otherwise, the builtin is left alone, and if it +reaches IRGen, IRGen should conservatively fold it to "1". In the common +case where `Array` is inlined and specialized, this will allow us to +eliminate all of the overhead in the important C cases. + +Opportunity Feature +------------------- + +For hardcore systems programming, we can expose `ContiguousArray` as a +user-consumable type. That will allow programmers who don't care about +Cocoa interoperability to avoid ever paying the cost of branching on +representation. This type would not bridge transparently to Array, but +could be useful if you need an array of Objective-C type, don't care +about NSArray compatibility, and care deeply about performance. + +Other Approaches Considered +--------------------------- + +We considered an approach where conversions between `NSArray` and native +Swift `Array` were entirely manual and quickly ruled it out as failing +to satisfy the requirements. + +We considered another promising proposal that would make `T[]` a +(hand-rolled) existential wrapper type. Among other things, we felt this +approach would expose multiple array types too prominently and would +tend to "bless" an inappropriately-specific protocol as the generic +collection interface (for example, a generic collection should not be +indexable with `Int`). + +We also considered several variants of the approach we've proposed here, +tuning the criteria by which we'd decide to optimize for a `Native` +representation. + +------------------------------------------------------------------------ + +[^1]: Value semantics dictates that when bridging an `NSArray` into + Swift, we invoke its `copy` method. Calling `copy` on an immutable + `NSArray` can be almost cost-free, but a mutable `NSArray` *will* be + physically copied. We accept that copy as the cost of doing + business. + +[^2]: Of course, by default, array bounds checking is enabled. C does + not include array bounds checks, so to get true C performance in all + cases, these will have to be disabled. diff --git a/docs/proposals/ArrayBridge.rst b/docs/proposals/ArrayBridge.rst deleted file mode 100644 index a3bc33009952c..0000000000000 --- a/docs/proposals/ArrayBridge.rst +++ /dev/null @@ -1,138 +0,0 @@ -:orphan: - -.. ===-- ArrayBridge.rst - Proposal for Bridging Swift Array and NSArray --===.. -.. -.. This source file is part of the Swift.org open source project -.. -.. Copyright (c) 2014 - 2015 Apple Inc. and the Swift project authors -.. Licensed under Apache License v2.0 with Runtime Library Exception -.. -.. See http://swift.org/LICENSE.txt for license information -.. See http://swift.org/CONTRIBUTORS.txt for the list of Swift project authors -.. -.. ===---------------------------------------------------------------------===.. - -===================================== - Bridging Swift Arrays to/from Cocoa -===================================== - -:Authors: Chris Lattner, Joe Groff, Dave Abrahams - -:Summary: Unifying a fast C-style array with a Cocoa class cluster - that can represent arbitrarily complex data structures is - challenging. In a space where no approach satisfies all - desires, we believe we've found a good compromise. - -Basic Requirements -================== - -A successfully-bridged array type would be both "great for Cocoa" and -"great for C." - -Being "great for Cocoa" means this must work and be efficient:: - - var a = [cocoaObject1, cocoaObject2] - someCocoaObject.takesAnNSArray(a) - - func processViews(views: AnyObject[]) { ... } - var b = someNSWindow.views // views is an NSArray - processViews(b) - - var c: AnyObject[] = someNSWindow.views - -Being "great For C" means that an array created in Swift must have -C-like performance and be representable as a base pointer and -length, for interaction with C APIs, at zero cost. - -Proposed Solution -================= - -``Array``, a.k.a. ``T[]``, is notionally an ``enum`` with two -cases; call them ``Native`` and ``Cocoa``. The ``Native`` case stores -a ``ContiguousArray``, which has a known, contiguous buffer -representation and O(1) access to the address of any element. The -``Cocoa`` case stores an ``NSArray``. - -``NSArray`` bridges bidirectionally in O(1) [#copy]_ to -``AnyObject[]``. It also implicitly converts in to ``T[]``, where T -is any class declared to be ``@objc``. No dynamic check of element -types is ever performed for arrays of ``@objc`` elements; instead we -simply let ``objc_msgSend`` fail when ``T``\ 's API turns out to be -unsupported by the object. Any ``T[]``, where T is an ``@objc`` -class, converts implicitly to NSArray. - -Optimization -============ - -Any type with more than one representation naturally penalizes -fine-grained operations such as indexing, because the cost of -repeatedly branching to handle each representation becomes -significant. For example, the design above would pose significant performance -problems for arrays of integers, because every subscript operation would have to -check to see if the representation is an NSArray, realize it is not, then do the -constant time index into the native representation. Beyond requiring an extra -check, this check would disable optimizations that can provide a significant -performance win (like auto-vectorization). - -However, the inherent limitations of ``NSArray`` mean that we can -often know at compile-time which representation is in play. So the -plan is to teach the compiler to optimize for the ``Native`` case -unless the element type is an ``@objc`` class or AnyObject. When ``T`` is -statically known not to be an ``@objc`` class or AnyObject, it will be -possible to eliminate the ``Cocoa`` case entirely. When generating code for -generic algorithms, we can favor the ``Native`` case, perhaps going so -far as to specialize for the case where all parameters are non-\ ``@objc`` -classes. This will give us C-like performance for array operations on ``Int``, -``Float``, and other ``struct`` types [#boundscheck]_. - -To implement this, we'll need to implement a new generic builtin, -something along the lines of "``Builtin.couldBeObjCType()``", which -returns a ``Builtin.Int1`` value. SILCombine and IRGen should eagerly -fold this to "0" iff ``T`` is known to be a protocol other than -AnyObject, if it is known to be a non-\ ``@objc`` class, or if it is -known to be any struct, enum or tuple. Otherwise, the builtin is left -alone, and if it reaches IRGen, IRGen should conservatively fold it to -"1". In the common case where ``Array`` is inlined and -specialized, this will allow us to eliminate all of the overhead in -the important C cases. - - -Opportunity Feature -=================== - -For hardcore systems programming, we can expose ``ContiguousArray`` as -a user-consumable type. That will allow programmers who don't care -about Cocoa interoperability to avoid ever paying the cost of -branching on representation. This type would not bridge transparently to Array, -but could be useful if you need an array of Objective-C type, don't care about -NSArray compatibility, and care deeply about performance. - -Other Approaches Considered -=========================== - -We considered an approach where conversions between ``NSArray`` and -native Swift ``Array`` were entirely manual and quickly ruled it out -as failing to satisfy the requirements. - -We considered another promising proposal that would make ``T[]`` a -(hand-rolled) existential wrapper type. Among other things, we felt -this approach would expose multiple array types too prominently and -would tend to "bless" an inappropriately-specific protocol as the -generic collection interface (for example, a generic collection should -not be indexable with ``Int``). - -We also considered several variants of the approach we've proposed -here, tuning the criteria by which we'd decide to optimize for a -``Native`` representation. - ---------- - -.. [#copy] Value semantics dictates that when bridging an ``NSArray`` - into Swift, we invoke its ``copy`` method. Calling ``copy`` on an - immutable ``NSArray`` can be almost cost-free, but a mutable - ``NSArray`` *will* be physically copied. We accept that copy as - the cost of doing business. - -.. [#boundscheck] Of course, by default, array bounds checking is enabled. - C does not include array bounds checks, so to get true C performance in all - cases, these will have to be disabled. diff --git a/docs/proposals/AttrC.md b/docs/proposals/AttrC.md new file mode 100644 index 0000000000000..ca42da38984d4 --- /dev/null +++ b/docs/proposals/AttrC.md @@ -0,0 +1,163 @@ +Eventually, we would like to write Swift modules which define pure-C +entry points for top-level functions, and be able to export more data +types to C code. + +This will be important for the Linux port, but also perhaps for system +frameworks that want to transition to Swift. + +The radars tracking this work are: + +- rdar://22488618 - @c top-level functions +- rdar://22490914 - @c structs + +The new @c attribute +==================== + +This attribute can be applied to the following kinds of declarations: + +- top-level functions +- static methods in non-generic classes +- enums +- structs + +There are two forms of the attribute: + + @c + @c(asmname) + +The latter allows the exported name to be set. By default, the exported +name is the unmangled, unqualified name of the function or nominal type. + +There is the question of how to gracefully handle name conflicts inside +a module. Since C does not have real modules or qualified names, we +probably can't catch name conflicts until link time. At the very least, +we will prohibit overloading `@c` functions (unless we use Clang-style +mangling and `__attribute__((overloadable))`). + +However, we might want to prefix the default @asmname of a @c symbol +with the Swift module name followed by an underscore, instead of using +the unqualified name. + +Type bridging +============= + +The rules for bridging types in `@c` function signatures are a subset of +`@objc`. + +Bridgeable types are now partitioned into two broad categories, "POD" +and "non-POD". POD types include: + +- integers +- floating point numbers +- @c enums +- fixed size arrays (currently presented as homogeneous tuples of + POD types) +- @c structs (whose fields must all be POD types) +- pointers to C types +- @convention(c) function types + +On Linux, we can't have reference counted pointers here at all, and +NSArray, etc do not exist, so only POD types are bridgeable. We must +ensure that we produce the right diagnostic and not crash when the user +references NSArray, etc on Linux. + +On Darwin, we can allow passing reference counted pointers directly as +function parameters. They are still not allowed as fields in `@c` +structs, though. + +The convention for arguments and results an be the same as +CoreFoundation functions imported from C. The code in +`CFunctionConventions` in SILFunctionType.cpp looks relevant. + +Bridging header output +====================== + +We can reuse most of `PrintAsObjC` to allow generating pure-C headers +for Swift modules which use @c but not @objc. + +Exporting functions to C +======================== + +Applying `@c` to a function is like a combination of `@convention(c)` +and `@asmname(func_name)`. + +The types in the function signature are bridged as described above, and +a foreign entry point is generated with the C calling convention and +given asmname. + +When the function is referenced from a `DeclRefExpr` inside of a +`FunctionConversionExpr` to `@convention(c)`, we emit a direct reference +to this foreign entry point. + +@c applied to enums and structs +=============================== + +For enums, `@c` and `@objc` can be synonyms. We still have to track +which one the user used, for accurate printing. On Linux, `@objc` could +probably be changed to always diagnose, but this will require changing +some tests. + +As stated above, all the fields of a `@c` struct must themselves be POD. + +Structs declared as `@c` need to be laid out with C size and alignment +conventions. We already do that for Swift structs imported from Clang by +asking Clang to do the layout on Clang AST, so perhaps for `@c` structs +declared in Swift, we can go in the other direction by constructing +Clang AST for the struct. + +Accessibility and linkage for @c declarations +============================================= + +Only public enums and structs should appear in generated headers; for +private types, `@c` only affects layout and restrictions on the field +types. + +For functions, it is not clear if private together with `@c` is useful, +but it could be implemented for completeness. We could either give the +foreign entry point private linkage, or intentionally give it incorrect +linkage allowing it to be found with `dlsym()`. + +inout parameters in @c and @objc functions +========================================== + +Right now we don't allow `inout` parameters for `@objc` methods. We +could export them as nonnull pointers, using +`__attribute__((nonnull(N)))` rather than `_Nonnull`. + +If we ever get as far as implementing C++ interoperability, we could +also export inouts as references rather than pointers. + +Diagnostics +=========== + +Right now all diagnostics for type bridging in Sema talk about +Objective-C, leading to funny phrasing when using invalid types in a +`@convention(c)` function, for instance. + +All diagnostics need to be audited to take the language as a parameter, +and either say `cannot be represented in C` or +`cannot be represented in Objective-C`. A Linux user should never see +diagnostics talking about Objective-C, except maybe if they explicitly +mention `@objc` in their code. + +On the plus side, it is okay if we conservatively talk about `C` in +Objective-C diagnostics on Darwin. + +Cleanups +======== + +Right now various aspects of the type bridging mapping are duplicated in +several places: + +- ASTContext::getBridgedToObjC() +- TypeChecker::isRepresentableInObjC() (various overloads) +- include/swift/ClangImporter/BuiltinMappedTypes.def +- include/swift/SIL/BridgedTypes.def +- TypeConverter::getLoweredCBridgedType() +- ClangImporter::VisitObjCObjectPointerType() and other places in + ImportType.cpp +- PrintAsObjC::printIfKnownGenericStruct() +- PrintAsObjC::printIfKnownTypeName() + +We should try to consolidate some of this if possible, to make the rules +more consistent and easier to describe between Darwin and Linux. diff --git a/docs/proposals/AttrC.rst b/docs/proposals/AttrC.rst deleted file mode 100644 index e633857eee7b3..0000000000000 --- a/docs/proposals/AttrC.rst +++ /dev/null @@ -1,163 +0,0 @@ -:orphan: - -Eventually, we would like to write Swift modules which define pure-C entry -points for top-level functions, and be able to export more data types to -C code. - -This will be important for the Linux port, but also perhaps for system -frameworks that want to transition to Swift. - -The radars tracking this work are: - -- rdar://22488618 - @c top-level functions -- rdar://22490914 - @c structs - -The new @c attribute -==================== - -This attribute can be applied to the following kinds of declarations: - -- top-level functions -- static methods in non-generic classes -- enums -- structs - -There are two forms of the attribute:: - - @c - @c(asmname) - -The latter allows the exported name to be set. By default, the exported -name is the unmangled, unqualified name of the function or nominal type. - -There is the question of how to gracefully handle name conflicts inside -a module. Since C does not have real modules or qualified names, we -probably can't catch name conflicts until link time. At the very least, -we will prohibit overloading ``@c`` functions (unless we use Clang-style -mangling and ``__attribute__((overloadable))``). - -However, we might want to prefix the default @asmname of a @c symbol -with the Swift module name followed by an underscore, instead of using -the unqualified name. - -Type bridging -============= - -The rules for bridging types in ``@c`` function signatures are a subset -of ``@objc``. - -Bridgeable types are now partitioned into two broad categories, "POD" -and "non-POD". POD types include: - -- integers -- floating point numbers -- @c enums -- fixed size arrays (currently presented as homogeneous tuples of POD types) -- @c structs (whose fields must all be POD types) -- pointers to C types -- @convention(c) function types - -On Linux, we can't have reference counted pointers here at all, and -NSArray, etc do not exist, so only POD types are bridgeable. We must -ensure that we produce the right diagnostic and not crash when the -user references NSArray, etc on Linux. - -On Darwin, we can allow passing reference counted pointers directly -as function parameters. They are still not allowed as fields in ``@c`` -structs, though. - -The convention for arguments and results an be the same as CoreFoundation -functions imported from C. The code in ``CFunctionConventions`` in -SILFunctionType.cpp looks relevant. - -Bridging header output -====================== - -We can reuse most of ``PrintAsObjC`` to allow generating pure-C headers -for Swift modules which use @c but not @objc. - -Exporting functions to C -======================== - -Applying ``@c`` to a function is like a combination of ``@convention(c)`` -and ``@asmname(func_name)``. - -The types in the function signature are bridged as described above, and a -foreign entry point is generated with the C calling convention and given -asmname. - -When the function is referenced from a ``DeclRefExpr`` inside of a -``FunctionConversionExpr`` to ``@convention(c)``, we emit a direct -reference to this foreign entry point. - -@c applied to enums and structs -=============================== - -For enums, ``@c`` and ``@objc`` can be synonyms. We still have to track -which one the user used, for accurate printing. On Linux, ``@objc`` -could probably be changed to always diagnose, but this will require -changing some tests. - -As stated above, all the fields of a ``@c`` struct must themselves be POD. - -Structs declared as ``@c`` need to be laid out with C size and alignment -conventions. We already do that for Swift structs imported from Clang by -asking Clang to do the layout on Clang AST, so perhaps for ``@c`` structs -declared in Swift, we can go in the other direction by constructing Clang -AST for the struct. - -Accessibility and linkage for @c declarations -============================================= - -Only public enums and structs should appear in generated headers; for -private types, ``@c`` only affects layout and restrictions on the field -types. - -For functions, it is not clear if private together with ``@c`` is useful, -but it could be implemented for completeness. We could either give the -foreign entry point private linkage, or intentionally give it incorrect -linkage allowing it to be found with ``dlsym()``. - -inout parameters in @c and @objc functions -========================================== - -Right now we don't allow ``inout`` parameters for ``@objc`` methods. -We could export them as nonnull pointers, using ``__attribute__((nonnull(N)))`` -rather than ``_Nonnull``. - -If we ever get as far as implementing C++ interoperability, we could also -export inouts as references rather than pointers. - -Diagnostics -=========== - -Right now all diagnostics for type bridging in Sema talk about Objective-C, -leading to funny phrasing when using invalid types in a ``@convention(c)`` -function, for instance. - -All diagnostics need to be audited to take the language as a parameter, and -either say ``cannot be represented in C`` or ``cannot be represented in -Objective-C``. A Linux user should never see diagnostics talking about -Objective-C, except maybe if they explicitly mention ``@objc`` in their code. - -On the plus side, it is okay if we conservatively talk about ``C`` in -Objective-C diagnostics on Darwin. - -Cleanups -======== - -Right now various aspects of the type bridging mapping are duplicated in -several places: - -- ASTContext::getBridgedToObjC() -- TypeChecker::isRepresentableInObjC() (various overloads) -- include/swift/ClangImporter/BuiltinMappedTypes.def -- include/swift/SIL/BridgedTypes.def -- TypeConverter::getLoweredCBridgedType() -- ClangImporter::VisitObjCObjectPointerType() and other places in ImportType.cpp -- PrintAsObjC::printIfKnownGenericStruct() -- PrintAsObjC::printIfKnownTypeName() - -We should try to consolidate some of this if possible, to make the -rules more consistent and easier to describe between Darwin and Linux. - diff --git a/docs/proposals/C Pointer Argument Interop.md b/docs/proposals/C Pointer Argument Interop.md new file mode 100644 index 0000000000000..4bdc8e7b067c0 --- /dev/null +++ b/docs/proposals/C Pointer Argument Interop.md @@ -0,0 +1,348 @@ +Summary +======= + +Pointer arguments are a fact of life in C and Cocoa, and there's no way +we're going to realistically annotate or wrap every API safely. However, +there are plenty of well-behaved APIs that use pointer arguments in +well-behaved ways that map naturally to Swift argument conventions, and +we should interact with those APIs in a natural, Swift-ish way. To do +so, I propose adding language and library facilities that enable the +following uses of pointer arguments: + +- Const pointer arguments `const int *`, including const pointers to + ObjC classes `NSFoo * const *`, can be used as "in" array arguments, + as `inout` scalar arguments, or as `UnsafeMutablePointer` arguments. +- Non-const pointer arguments to C types, `int *`, can be used as + `inout` array or scalar arguments, or as + `UnsafeMutablePointer` arguments. +- Non-const pointer arguments to ObjC class types, `NSFoo **`, can be + used as `inout` scalar arguments or passed `nil`. (They cannot be + used as array arguments or as `UnsafeMutablePointer` arguments.) +- `const void *` and `void *` pointers can be used in the same ways as + pointers to any C type (but not ObjC types). + +This model intentionally doesn't try to provide a mapping to every +possible use case for pointers in C. It also intentionally avoids +addressing special cases we could potentially provide higher-level +support for. Some particular issues this proposal specifically does not +address: + +- Pointer return values +- Special handling of `char*` and/or `const char*` arguments +- Special handling of Core Foundation types +- Special handling of `NSError**` arguments +- Precise lifetime of values (beyond the minimal lifetime extension to + the duration of a call) +- Overriding of ObjC methods that take pointer arguments with subclass + methods that take non-pointer arguments + +Design Considerations +===================== + +Const Pointer Arguments +----------------------- + +### Arrays + +Const pointer arguments are frequently used in both C and Objective-C to +take an array of arguments effectively by value. To support this use +case, we should support passing a Swift `Array` value to a const pointer +argument. An example from Core Foundation is `CGColorCreate`, which +takes a `CGFloat` array of color space-specific components: + + let rgb = CGColorSpaceCreateCalibratedRGB() + let cyan = CGColorCreate(rgb, [0, 1, 1]) + +We are willing to assume that the API is well-behaved and does not +mutate the pointed-to array, so we can safely pass an interior pointer +to the array storage without worrying about mutation. We only guarantee +the lifetime of the array for the duration of the call, so it could +potentially be promoted to a stack allocation. + +### "In" Arguments + +Const pointer arguments are also used in many cases where a value is +unmodified, but its identity is important. Somewhat more rarely, const +pointer arguments are used as pure "in" value arguments with no regard +for identity; this is particularly prevalent on platforms like Win32 +where there has historically been no standard ABI for passing structs by +value, but pure "in" pointer parameters are rare on our platforms. The +potential consequences of disregarding value identity with C APIs are +too high to allow scalar arguments to implicitly be used as pointer +arguments: + + // From C: void foo(const pthread_mutex_t *); + import Foo + + let mutex = pthread_mutex_create() + // This would pass a different temporary buffer on each call--not what you + // want for a mutex! + foo(mutex) + foo(mutex) + foo(mutex) + +Although const pointers should never be used for actual mutation, we +propose that only `inout` scalar arguments be accepted for const pointer +parameters. Although our semantics normally do not guarantee value +identity, `inout` parameters that refer to stored variables or stored +properties of C-derived types are in practice never subjected to +implicit writebacks except in limited circumstances such as capture of +`inout` references in closures that could be diagnosed. Requiring +`inout` also prevents the use of rvalues or `let` bindings that never +have well-defined addresses as pointer arguments. This more clearly +communicates the intent for the callee to receive the same variable on +every call: + + // From C: void foo(const pthread_mutex_t *); + import Foo + + var mutex = pthread_mutex_create() + foo(&mutex) + foo(&mutex) + foo(&mutex) + +If using an rvalue as a pointer argument is desired, it can easily be +wrapped in an array. This communicates that the value *is* being copied +into the temporary array, so it's more obvious that identity would not +be maintained: + + // an immutable scalar we might want to pass into a "const double*". + let grayLevel = 0.5 + let monochrome = CGColorSpaceCreateGrayscale() + + // error, can't pass Double into second argument. + let c1 = CGColorCreate(monochrome, grayval) + // error, can't take the address of a 'let' (would be ok for a 'var') + let c2 = CGColorCreate(monochrome, &grayval) + // OK, we're explicitly forming an array + let c3 = CGColorCreate(monochrome, [grayval]) + +Non-Const Pointer Arguments +--------------------------- + +### C Types + +Non-const arguments of C type can be used as "out" or "inout" +parameters, either of scalars or of arrays, and so should accept `inout` +parameters of array or scalar type. Although a C API may expect a pure +"out" parameter and not require initialization of its arguments, it is +safer to assume the argument is `inout` and always require +initialization: + + var s, c: Double + // error, 's' and 'c' aren't initialized + sincos(0.5, &s, &c) + + var s1 = 0.0, c1 = 0.0 + // OK + sincos(0.5, &s1, &c1) + +For array parameters, the exact point of mutation inside the callee +cannot be known, so a copy-on-write array buffer must be eagerly uniqued +prior to the address of the array being taken: + + func loadFloatsFromData(data: NSData) { + var a: Float[] = [0.0, 0.0, 0.0, 0.0] + var b = a + + // Should only mutate 'b' without affecting 'a', so its backing store + // must be uniqued + data.getBytes(&b, sizeof(Float.self) * b.count) + } + +### ObjC Types + +ARC semantics treat an `NSFoo**` type as a pointer to an +`__autoreleasing` `NSFoo*`. Although in theory these interfaces could +receive arrays of object pointers in Objective-C, that use case doesn't +come up in Cocoa, and we can't reliably bridge such APIs into Swift. We +only need to bridge ObjC mutable pointer types to accept a scalar +`inout` object reference or `nil`. + +Pointer Return Values +--------------------- + +This proposal does not address the handling of return values, which +should still be imported into Swift as `UnsafeMutablePointer` values. + +Library Features +================ + +The necessary conversions can be represented entirely in the standard +library with the help of some new language features, inout address +conversion, inout writeback conversion, and interior pointer conversion, +described below. There are three categories of argument behavior needed, +and thus three new types. These types should have no user-accessible +operations of their own other than their implicit conversions. The +necessary types are as follows: + +- `CConstPointer` is the imported representation of a + `const T *` argument. It is implicitly convertible from `inout T` by + inout address conversion and from `Array` by immutable interior + pointer conversion. It is also implicitly convertible to and from + `UnsafeMutablePointer` by normal conversion. +- `CMutablePointer` is the imported representation of a `T *` + argument for a POD C type `T`. It is implicitly convertible from + `inout T` by inout address conversion and from `inout Array` by + mutating interior pointer conversion. It is also implicitly + convertible to and from `UnsafeMutablePointer` by + normal conversion. +- `ObjCInOut` is the imported representation of a `T **` argument + for an ObjC class type `T`. It is implicitly convertible from + `inout T` by inout writeback conversion and is implicitly + convertible from `nil`. It cannot be converted from an array or to + `UnsafeMutablePointer`. + +New Language Features +===================== + +To support the necessary semantics for argument passing, some new +conversion forms need to be supported by the language with special-cased +lifetime behavior. + +Interior Pointer Conversions +---------------------------- + +To be able to pass a pointer to array data as an argument, we need to be +able to guarantee the lifetime of the array buffer for the duration of +the call. If mutation can potentially occur through the pointer, then +copy-on-write buffers must also be uniqued prior to taking the address. +A new form of conversion, `@unsafe_interior_pointer_conversion`, can be +applied to an instance method of a type, to allow that type to return +both a converted pointer and an owning reference that guarantees the +validity of the pointer. Such methods can be either `mutating` or +non-mutating; only non-mutating conversions are considered for non- +`inout` parameters, and only `mutating` conversions are considered for +`inout` parameters: + + extension Array { + @unsafe_interior_pointer_conversion + func convertToConstPointer() + -> (CConstPointer, ArrayBuffer) { + return (CConstPointer(self.base), self.owner) + } + + @unsafe_interior_pointer_conversion + mutating func convertToMutablePointer() + -> (CMutablePointer, ArrayBuffer) { + // Make the backing buffer unique before handing out a mutable pointer. + self.makeUnique() + return (CMutablePointer(self.base), self.owner) + } + } + +`@unsafe_interior_pointer_conversion` conversions are only considered in +argument contexts. If such a conversion is found, the first element of +the return tuple is used as the argument, and a strong reference to the +second element is held for the duration of the callee that receives the +converted argument. + +Inout Address Conversion +------------------------ + +To pass an `inout` as a pointer argument, we need to be able to lock an +address for the `inout` for the duration of the call, which is not +normally possible. This functionality only needs to be available to the +standard library, so can be expressed in terms of builtins. A type can +conform to the `_BuiltinInOutAddressConvertible` protocol to be +convertible from an inout reference. The protocol is defined as follows: + + protocol _BuiltinInOutAddressConvertible { + /// The type from which inout conversions are allowed to the conforming + /// type. + typealias InOutType + + /// Create a value of the conforming type using the address of an inout + /// argument. + class func _convertFromInOutAddress(p: Builtin.RawPointer) -> Self + } + +An example of a conformance for `CMutablePointer`: + + struct CMutablePointer: _BuiltinInOutAddressConvertible { + let ptr: Builtin.RawPointer + + typealias InOutType = T + + @_transparent + static func _convertFromInOutAddress(p: Builtin.RawPointer) + -> CMutablePointer { + return CMutablePointer(p) + } + } + + func foo(p: CMutablePointer) { } + + var i = 0 + foo(&i) + +The lifetime of the variable, stored property owning object, or +writeback buffer backing the inout is guaranteed for the lifetime of the +callee that receives the converted parameter, as if the callee had +received the inout parameter directly. + +Inout Writeback Conversion +-------------------------- + +Inout address conversion alone is not enough for `ObjCInOut` to work as +intended, because the change to the `__autoreleasing` convention for the +pointed-to object reference requires a writeback temporary. The +`_BuiltinInOutWritebackConvertible` protocol allows for an additional +writeback to be introduced before and after the address of the `inout` +is taken: + + protocol _BuiltinInOutWritebackConvertible { + /// The original type from which inout conversions are allowed to the + /// conforming type. + typealias InOutType + + /// The type of the temporary writeback whose address is used to construct + /// the converted value. + typealias WritebackType + + /// Get the initial value the writeback temporary should have on entry to + /// the call. + class func _createWriteback(inout InOutType) -> WritebackType + + /// Create a value of the conforming type using the address of the writeback + /// temporary. + class func _convertFromWritebackAddress(p: Builtin.RawPointer) -> Self + + /// Write the writeback temporary back to the original value. + class func _commitWriteback(inout InOutType, WritebackType) + } + +An example of a conformance for `ObjCInOut`: + + struct ObjCInOut: _BuiltinInOutWritebackConvertible { + let ptr: Builtin.RawPointer + + typealias InOutType = T! + typealias WritebackType = Builtin.RawPointer + + @_transparent + static func _createWriteback(inout ref: T!) + -> Builtin.RawPointer { + // The initial object reference is passed into the callee effectively + // __unsafe_unretained, so pass it as a RawPointer. + return unsafeBitCast(ref, Builtin.RawPointer.self) + } + + @_transparent + static func _commitWriteback(inout ref: T!, + value: Builtin.RawPointer) { + // The reference is autoreleased on return from the caller, so retain it + // by loading it back as a T?. + ref = unsafeBitCast(value, T!.self) + } + + @_transparent + static func _convertFromWritebackAddress(value: Builtin.RawPointer) { + return ObjCInOut(value) + } + } + +The lifetime of the writeback is guaranteed for the lifetime of the +callee that receives the converted parameter, as if the callee had +received the writeback temporary as a mutable logical property of the +original inout parameter. diff --git a/docs/proposals/C Pointer Argument Interop.rst b/docs/proposals/C Pointer Argument Interop.rst deleted file mode 100644 index 0be9dd9f19cbf..0000000000000 --- a/docs/proposals/C Pointer Argument Interop.rst +++ /dev/null @@ -1,343 +0,0 @@ -:orphan: - -Summary -======= - -Pointer arguments are a fact of life in C and Cocoa, and there's no way we're -going to realistically annotate or wrap every API safely. However, there are -plenty of well-behaved APIs that use pointer arguments in well-behaved ways -that map naturally to Swift argument conventions, and we should interact with -those APIs in a natural, Swift-ish way. To do so, I propose adding language -and library facilities that enable the following uses of pointer -arguments: - -- Const pointer arguments ``const int *``, including const pointers to ObjC - classes ``NSFoo * const *``, can be used as "in" array arguments, - as ``inout`` scalar arguments, or as ``UnsafeMutablePointer`` arguments. -- Non-const pointer arguments to C types, ``int *``, can be used as ``inout`` - array or scalar arguments, or as ``UnsafeMutablePointer`` arguments. -- Non-const pointer arguments to ObjC class types, ``NSFoo **``, can be used as - ``inout`` scalar arguments or passed ``nil``. (They cannot be used as - array arguments or as ``UnsafeMutablePointer`` arguments.) -- ``const void *`` and ``void *`` pointers can be used in the same ways as - pointers to any C type (but not ObjC types). - -This model intentionally doesn't try to provide a mapping to every possible -use case for pointers in C. It also intentionally avoids addressing special -cases we could potentially provide higher-level support for. Some particular -issues this proposal specifically does not address: - -- Pointer return values -- Special handling of ``char*`` and/or ``const char*`` arguments -- Special handling of Core Foundation types -- Special handling of ``NSError**`` arguments -- Precise lifetime of values (beyond the minimal lifetime extension to the - duration of a call) -- Overriding of ObjC methods that take pointer arguments with subclass methods - that take non-pointer arguments - -Design Considerations -===================== - -Const Pointer Arguments ------------------------ - -Arrays -~~~~~~ - -Const pointer arguments are frequently used in both C and Objective-C to take -an array of arguments effectively by value. To support this use case, we should -support passing a Swift ``Array`` value to a const pointer argument. An -example from Core Foundation is ``CGColorCreate``, which takes a -``CGFloat`` array of color space-specific components:: - - let rgb = CGColorSpaceCreateCalibratedRGB() - let cyan = CGColorCreate(rgb, [0, 1, 1]) - -We are willing to assume that the API is well-behaved and does not mutate the -pointed-to array, so we can safely pass an interior pointer to the array storage -without worrying about mutation. We only guarantee the lifetime of the -array for the duration of the call, so it could potentially be promoted to a -stack allocation. - -"In" Arguments -~~~~~~~~~~~~~~ - -Const pointer arguments are also used in many cases where a value is unmodified, -but its identity is important. Somewhat more rarely, const pointer arguments -are used as pure "in" value arguments with no regard for identity; this is -particularly prevalent on platforms like Win32 where there has historically -been no standard ABI for passing structs by value, but pure "in" pointer -parameters are rare on our platforms. The potential consequences of -disregarding value identity with C APIs are too high to allow scalar arguments -to implicitly be used as pointer arguments:: - - // From C: void foo(const pthread_mutex_t *); - import Foo - - let mutex = pthread_mutex_create() - // This would pass a different temporary buffer on each call--not what you - // want for a mutex! - foo(mutex) - foo(mutex) - foo(mutex) - -Although const pointers should never be used for actual mutation, we propose -that only ``inout`` scalar arguments be accepted for const pointer parameters. -Although our semantics normally do not guarantee value identity, ``inout`` -parameters that refer to stored variables or stored properties of C-derived -types are in practice never subjected to implicit writebacks except in limited -circumstances such as capture of ``inout`` references in closures that could be -diagnosed. Requiring ``inout`` also prevents the use of rvalues or ``let`` -bindings that never have well-defined addresses as pointer arguments. This -more clearly communicates the intent for the callee to receive the same -variable on every call:: - - // From C: void foo(const pthread_mutex_t *); - import Foo - - var mutex = pthread_mutex_create() - foo(&mutex) - foo(&mutex) - foo(&mutex) - -If using an rvalue as a pointer argument is desired, it can easily be wrapped -in an array. This communicates that the value *is* being copied into the -temporary array, so it's more obvious that identity would not be maintained:: - - // an immutable scalar we might want to pass into a "const double*". - let grayLevel = 0.5 - let monochrome = CGColorSpaceCreateGrayscale() - - // error, can't pass Double into second argument. - let c1 = CGColorCreate(monochrome, grayval) - // error, can't take the address of a 'let' (would be ok for a 'var') - let c2 = CGColorCreate(monochrome, &grayval) - // OK, we're explicitly forming an array - let c3 = CGColorCreate(monochrome, [grayval]) - -Non-Const Pointer Arguments ---------------------------- - -C Types -~~~~~~~ - -Non-const arguments of C type can be used as "out" or "inout" parameters, -either of scalars or of arrays, and so should accept ``inout`` parameters of -array or scalar type. Although a C API may expect a pure "out" parameter and -not require initialization of its arguments, it is safer to assume the argument -is ``inout`` and always require initialization:: - - var s, c: Double - // error, 's' and 'c' aren't initialized - sincos(0.5, &s, &c) - - var s1 = 0.0, c1 = 0.0 - // OK - sincos(0.5, &s1, &c1) - -For array parameters, the exact point of mutation inside the callee cannot be -known, so a copy-on-write array buffer must be eagerly uniqued prior to the -address of the array being taken:: - - func loadFloatsFromData(data: NSData) { - var a: Float[] = [0.0, 0.0, 0.0, 0.0] - var b = a - - // Should only mutate 'b' without affecting 'a', so its backing store - // must be uniqued - data.getBytes(&b, sizeof(Float.self) * b.count) - } - -ObjC Types -~~~~~~~~~~ - -ARC semantics treat an ``NSFoo**`` type as a pointer to an ``__autoreleasing`` -``NSFoo*``. Although in theory these interfaces could receive arrays of object -pointers in Objective-C, that use case doesn't come up in Cocoa, and we can't -reliably bridge such APIs into Swift. We only need to bridge ObjC mutable pointer -types to accept a scalar ``inout`` object reference or ``nil``. - -Pointer Return Values ---------------------- - -This proposal does not address the handling of return values, which should still -be imported into Swift as ``UnsafeMutablePointer`` values. - - -Library Features -================ - -The necessary conversions can be represented entirely in the standard library -with the help of some new language features, inout address conversion, inout -writeback conversion, and interior pointer conversion, described below. There -are three categories of argument behavior needed, and thus three new types. -These types should have no user-accessible operations of their own other than -their implicit conversions. The necessary types are as follows: - -- ``CConstPointer`` is the imported representation of a ``const T *`` - argument. It is implicitly convertible from ``inout T`` by inout address - conversion and from ``Array`` by immutable interior pointer - conversion. It is also implicitly convertible to and from ``UnsafeMutablePointer`` - by normal conversion. -- ``CMutablePointer`` is the imported representation of a ``T *`` - argument for a POD C type ``T``. It is implicitly convertible from - ``inout T`` by inout address conversion and from ``inout Array`` by mutating - interior pointer conversion. It is also implicitly convertible to and from - ``UnsafeMutablePointer`` by normal conversion. -- ``ObjCInOut`` is the imported representation of a ``T **`` - argument for an ObjC class type ``T``. It is implicitly convertible from - ``inout T`` by inout writeback conversion and is implicitly convertible - from ``nil``. It cannot be converted from an array or to ``UnsafeMutablePointer``. - -New Language Features -===================== - -To support the necessary semantics for argument passing, some new conversion -forms need to be supported by the language with special-cased lifetime behavior. - -Interior Pointer Conversions ----------------------------- - -To be able to pass a pointer to array data as an argument, we need to be able -to guarantee the lifetime of the array buffer for the duration of the call. -If mutation can potentially occur through the pointer, then copy-on-write -buffers must also be uniqued prior to taking the address. A new form of -conversion, ``@unsafe_interior_pointer_conversion``, can be applied to an -instance method of a type, to allow that type to return both a converted -pointer and an owning reference that guarantees the validity of the pointer. -Such methods can be either ``mutating`` or non-mutating; only non-mutating -conversions are considered for non- ``inout`` parameters, and only ``mutating`` -conversions are considered for ``inout`` parameters:: - - extension Array { - @unsafe_interior_pointer_conversion - func convertToConstPointer() - -> (CConstPointer, ArrayBuffer) { - return (CConstPointer(self.base), self.owner) - } - - @unsafe_interior_pointer_conversion - mutating func convertToMutablePointer() - -> (CMutablePointer, ArrayBuffer) { - // Make the backing buffer unique before handing out a mutable pointer. - self.makeUnique() - return (CMutablePointer(self.base), self.owner) - } - } - -``@unsafe_interior_pointer_conversion`` conversions are only considered in -argument contexts. If such a conversion is found, the first element of the -return tuple is used as the argument, and a strong reference to the second -element is held for the duration of the callee that receives the converted -argument. - -Inout Address Conversion ------------------------- - -To pass an ``inout`` as a pointer argument, we need to be able to lock an -address for the ``inout`` for the duration of the call, which is not normally -possible. This functionality only needs to be available to the standard library, -so can be expressed in terms of builtins. A type can conform to the -``_BuiltinInOutAddressConvertible`` protocol to be convertible from an -inout reference. The protocol is defined as follows:: - - protocol _BuiltinInOutAddressConvertible { - /// The type from which inout conversions are allowed to the conforming - /// type. - typealias InOutType - - /// Create a value of the conforming type using the address of an inout - /// argument. - class func _convertFromInOutAddress(p: Builtin.RawPointer) -> Self - } - -An example of a conformance for ``CMutablePointer``:: - - struct CMutablePointer: _BuiltinInOutAddressConvertible { - let ptr: Builtin.RawPointer - - typealias InOutType = T - - @_transparent - static func _convertFromInOutAddress(p: Builtin.RawPointer) - -> CMutablePointer { - return CMutablePointer(p) - } - } - - func foo(p: CMutablePointer) { } - - var i = 0 - foo(&i) - -The lifetime of the variable, stored property owning object, or writeback -buffer backing the inout is guaranteed for the lifetime of the callee that -receives the converted parameter, as if the callee had received the inout -parameter directly. - -Inout Writeback Conversion --------------------------- - -Inout address conversion alone is not enough for ``ObjCInOut`` to work as -intended, because the change to the ``__autoreleasing`` convention for the -pointed-to object reference requires a writeback temporary. The -``_BuiltinInOutWritebackConvertible`` protocol allows for an additional -writeback to be introduced before and after the address of the ``inout`` is -taken:: - - protocol _BuiltinInOutWritebackConvertible { - /// The original type from which inout conversions are allowed to the - /// conforming type. - typealias InOutType - - /// The type of the temporary writeback whose address is used to construct - /// the converted value. - typealias WritebackType - - /// Get the initial value the writeback temporary should have on entry to - /// the call. - class func _createWriteback(inout InOutType) -> WritebackType - - /// Create a value of the conforming type using the address of the writeback - /// temporary. - class func _convertFromWritebackAddress(p: Builtin.RawPointer) -> Self - - /// Write the writeback temporary back to the original value. - class func _commitWriteback(inout InOutType, WritebackType) - } - -An example of a conformance for ``ObjCInOut``:: - - struct ObjCInOut: _BuiltinInOutWritebackConvertible { - let ptr: Builtin.RawPointer - - typealias InOutType = T! - typealias WritebackType = Builtin.RawPointer - - @_transparent - static func _createWriteback(inout ref: T!) - -> Builtin.RawPointer { - // The initial object reference is passed into the callee effectively - // __unsafe_unretained, so pass it as a RawPointer. - return unsafeBitCast(ref, Builtin.RawPointer.self) - } - - @_transparent - static func _commitWriteback(inout ref: T!, - value: Builtin.RawPointer) { - // The reference is autoreleased on return from the caller, so retain it - // by loading it back as a T?. - ref = unsafeBitCast(value, T!.self) - } - - @_transparent - static func _convertFromWritebackAddress(value: Builtin.RawPointer) { - return ObjCInOut(value) - } - } - -The lifetime of the writeback is guaranteed for the lifetime of the callee that -receives the converted parameter, as if the callee had received the writeback -temporary as a mutable logical property of the original inout parameter. - diff --git a/docs/proposals/C Pointer Interop Language Model.md b/docs/proposals/C Pointer Interop Language Model.md new file mode 100644 index 0000000000000..155093b99ee85 --- /dev/null +++ b/docs/proposals/C Pointer Interop Language Model.md @@ -0,0 +1,212 @@ +We have a pretty good user model for C pointer interop now, but the +language model still needs improvement. Building the user model on top +of implicit conversions has a number of undesirable side effects. We end +up with a mess of pointer types—the intended user-facing, one-word +pointer types `UnsafeMutablePointer` and `COpaquePointer`, which expose +a full pointer-ish API and are naturally ABI-compatible with C pointers; +and the bridging pointer types, `ObjCMutablePointer`, `CMutablePointer`, +`CConstPointer`, `CMutableVoidPointer`, and `CConstVoidPointer`, which +have no real API yet but exist only to carry an owner reference and be +implicitly convertible, and rely on compiler magic to be passed to C +functions. Since we can do the magic pointer bridging only in limited +places, we assault users writing method overrides and reading +synthesized headers with both sets of pointer types in a confusing +jumble. + +The best solution to this is to burn the user model into the language, +giving function applications special powers to provide the user model +for pointers. We then provide only one set of plain pointer types, with +special intrinsic behavior when used as function arguments. + +The Pointer Types +================= + +In the standard library, we provide three pointer types: + +- `UnsafePointer`, corresponding to `T const *` in C and ARC, +- `UnsafeMutablePointer`, corresponding to `T *` in C, and + `T* __strong *` in ARC for class types, and +- `AutoreleasingUnsafeMutablePointer` (for all `T: AnyObject`), + corresponding to `T* __autoreleasing *` in ARC. + +These types are all one word, have no ownership semantics, and share a +common interface. `UnsafePointer` does not expose operations for storing +to the referenced memory. `UnsafeMutablePointer` and +`AutoreleasingUnsafeMutablePointer` differ in store behavior: +`UnsafeMutablePointer` assumes that the pointed-to reference has +ownership semantics, so `ptr.initialize(x)` consumes a reference to `x`, +and `ptr.assign(x)` releases the originally stored value before storing +the new value. `AutoreleasingUnsafeMutablePointer` assumes that the +pointed-to reference does not have ownership semantics, so values are +autoreleased before being stored by either initialize() or assign(), and +no release is done on reassignment. Loading from any of the three kinds +of pointer does a strong load, so there is no need for a separate +`AutoreleasingUnsafePointer`. + +Conversion behavior for pointer arguments +========================================= + +The user model for pointer arguments becomes an inherent capability of +function applications. The rules are: + +UnsafeMutablePointer<T> +----------------------------- + +When a function is declared as taking an `UnsafeMutablePointer` +argument, it can accept any of the following: + +- `nil`, which is passed as a null pointer, +- an `UnsafeMutablePointer` value, which is passed verbatim, +- an inout expression whose operand is a stored lvalue of type `T`, + which is passed as the address of the lvalue, or +- an inout `Array` value, which is passed as a pointer to the start + of the array, and lifetime-extended for the duration of the callee. + +As a special case, when a function is declared as taking an +`UnsafeMutablePointer` argument, it can accept the same operands +as `UnsafeMutablePointer` for any type T. + +So if you have a function declared: + + func foo(x: UnsafeMutablePointer) + +You can call it as any of: + + var x: Float = 0.0 + var p: UnsafeMutablePointer = nil + var a: Float[] = [1.0, 2.0, 3.0] + foo(nil) + foo(p) + foo(&x) + foo(&a) + +And if you have a function declared: + + func bar(x: UnsafeMutablePointer) + +You can call it as any of: + + var x: Float = 0.0, y: Int = 0 + var p: UnsafeMutablePointer = nil, q: UnsafeMutablePointer = nil + var a: Float[] = [1.0, 2.0, 3.0], b: Int = [1, 2, 3] + bar(nil) + bar(p) + bar(q) + bar(&x) + bar(&y) + bar(&a) + bar(&b) + +AutoreleasingUnsafeMutablePointer<T> +------------------------------------------ + +When a function is declared as taking an +`AutoreleasingUnsafeMutablePointer`, it can accept any of the +following: + +- nil, which is passed as a null pointer, +- an `AutoreleasingUnsafeMutablePointer` value, which is passed + verbatim, or +- an inout expression, whose operand is primitive-copied to a + temporary nonowning buffer. The address of that buffer is passed to + the callee, and on return, the value in the buffer is loaded, + retained, and reassigned into the operand. + +Note that the above list does not include arrays, since implicit +autoreleasing-to-strong writeback of an entire array would probably not +be desirable. + +So if you have a function declared: + + func bas(x: AutoreleasingUnsafeMutablePointer) + +You can call it as any of: + + var x: NSBas? = nil + var p: AutoreleasingUnsafeMutablePointer = nil + bas(nil) + bas(p) + bas(&x) + +UnsafePointer<T> +---------------------- + +When a function is declared as taking an `UnsafeMutablePointer` +argument, it can accept any of the following: + +- nil, which is passed as a null pointer, +- an `UnsafeMutablePointer`, `UnsafePointer`, or + `AutoreleasingUnsafeMutablePointer` value, which is converted to + `UnsafePointer` if necessary and passed verbatim, +- an inout expression whose operand is an lvalue of type `T`, which is + passed as the address of (the potentially temporary writeback + buffer of) the lvalue, or +- an `Array` value, which is passed as a pointer to the start of + the array, and lifetime-extended for the duration of the callee. + +As a special case, when a function is declared as taking an +`UnsafePointer` argument, it can accept the same operands as +`UnsafePointer` for any type `T`. Pointers to certain integer types +can furthermore interoperate with strings; see Strings\_ below. + +So if you have a function declared: + + func zim(x: UnsafePointer) + +You can call it as any of: + + var x: Float = 0.0 + var p: UnsafePointer = nil + zim(nil) + zim(p) + zim(&x) + zim([1.0, 2.0, 3.0]) + +And if you have a function declared: + + func zang(x: UnsafePointer) + +You can call it as any of: + + var x: Float = 0.0, y: Int = 0 + var p: UnsafePointer = nil, q: UnsafePointer = nil + zang(nil) + zang(p) + zang(q) + zang(&x) + zang(&y) + let doubles = [1.0, 2.0, 3.0] + let ints = [1, 2, 3] + zang(doubles) + zang(ints) + +A type checker limitation prevents array literals from being passed +directly to `UnsafePointer` arguments without type annotation. As +a workaround, you can bind the array literal to a constant, as above, or +specify the array type with `as`: + + zang([1.0, 2.0, 3.0] as [Double]) + zang([1, 2, 3] as [Int]) + +This limitation is tracked as <rdar://problem/17444930>. + +Strings +======= + +Pointers to the following C integer and character types can interoperate +with Swift `String` values and string literals: + +- `CChar`, `CSignedChar`, and `CUnsignedChar`, which interoperate with + `String` as a UTF-8 code unit array; +- (not implemented yet) `CShort`, `CUnsignedShort`, and `CChar16`, + which interoperate with `String` as a UTF-16 code unit array; and +- (not implemented yet) `CInt`, `CUnsignedInt`, `CWideChar`, and + `CChar32`, which interoperate with `String` as a UTF-32 code + unit array. + +A `UnsafePointer` parameter with any of the above element types may take +a `String` value as an argument. The string is transcoded to a +null-terminated buffer of the appropriate encoding, if necessary, and a +pointer to the buffer is passed to the function. The callee may not +mutate through the array, and the referenced memory is only guaranteed +to live for the duration of the call. diff --git a/docs/proposals/C Pointer Interop Language Model.rst b/docs/proposals/C Pointer Interop Language Model.rst deleted file mode 100644 index aa51aef2b0d89..0000000000000 --- a/docs/proposals/C Pointer Interop Language Model.rst +++ /dev/null @@ -1,205 +0,0 @@ -:orphan: - -We have a pretty good user model for C pointer interop now, but the language -model still needs improvement. Building the user model on top of implicit -conversions has a number of undesirable side effects. We end up with a mess of -pointer types—the intended user-facing, one-word pointer types -``UnsafeMutablePointer`` and ``COpaquePointer``, which expose a full pointer-ish API -and are naturally ABI-compatible with C pointers; and the bridging pointer -types, ``ObjCMutablePointer``, ``CMutablePointer``, ``CConstPointer``, -``CMutableVoidPointer``, and ``CConstVoidPointer``, which have no real API yet -but exist only to carry an owner reference and be implicitly convertible, and -rely on compiler magic to be passed to C functions. Since we can do the magic -pointer bridging only in limited places, we assault users writing method -overrides and reading synthesized headers with both sets of pointer types in a -confusing jumble. - -The best solution to this is to burn the user model into the language, giving -function applications special powers to provide the user model for pointers. We -then provide only one set of plain pointer types, with -special intrinsic behavior when used as function arguments. - -The Pointer Types -================= - -In the standard library, we provide three pointer types: - -- ``UnsafePointer``, corresponding to ``T const *`` in C and ARC, -- ``UnsafeMutablePointer``, corresponding to ``T *`` in C, and ``T* __strong *`` in - ARC for class types, and -- ``AutoreleasingUnsafeMutablePointer`` (for all ``T: AnyObject``), corresponding - to ``T* __autoreleasing *`` in ARC. - -These types are all one word, have no ownership semantics, and share a common -interface. ``UnsafePointer`` does not expose operations for storing to the -referenced memory. ``UnsafeMutablePointer`` and ``AutoreleasingUnsafeMutablePointer`` differ -in store behavior: ``UnsafeMutablePointer`` assumes that the pointed-to reference has -ownership semantics, so ``ptr.initialize(x)`` consumes a reference to ``x``, -and ``ptr.assign(x)`` releases the originally stored value before storing the -new value. ``AutoreleasingUnsafeMutablePointer`` assumes that the pointed-to -reference does not have ownership semantics, so values are autoreleased before -being stored by either initialize() or assign(), and no release is done on -reassignment. Loading from any of the three kinds of pointer does a strong -load, so there is no need for a separate ``AutoreleasingUnsafePointer``. - -Conversion behavior for pointer arguments -========================================= - -The user model for pointer arguments becomes an inherent capability of function applications. The rules are: - -UnsafeMutablePointer ------------------------ - -When a function is declared as taking an ``UnsafeMutablePointer`` argument, it can -accept any of the following: - -- ``nil``, which is passed as a null pointer, -- an ``UnsafeMutablePointer`` value, which is passed verbatim, -- an inout expression whose operand is a stored lvalue of type ``T``, which is - passed as the address of the lvalue, or -- an inout ``Array`` value, which is passed as a pointer to the start of the - array, and lifetime-extended for the duration of the callee. - -As a special case, when a function is declared as taking an -``UnsafeMutablePointer`` argument, it can accept the same operands as -``UnsafeMutablePointer`` for any type T. - -So if you have a function declared:: - - func foo(x: UnsafeMutablePointer) - -You can call it as any of:: - - var x: Float = 0.0 - var p: UnsafeMutablePointer = nil - var a: Float[] = [1.0, 2.0, 3.0] - foo(nil) - foo(p) - foo(&x) - foo(&a) - -And if you have a function declared:: - - func bar(x: UnsafeMutablePointer) - -You can call it as any of:: - - var x: Float = 0.0, y: Int = 0 - var p: UnsafeMutablePointer = nil, q: UnsafeMutablePointer = nil - var a: Float[] = [1.0, 2.0, 3.0], b: Int = [1, 2, 3] - bar(nil) - bar(p) - bar(q) - bar(&x) - bar(&y) - bar(&a) - bar(&b) - -AutoreleasingUnsafeMutablePointer ------------------------------------- - -When a function is declared as taking an ``AutoreleasingUnsafeMutablePointer``, it -can accept any of the following: - -- nil, which is passed as a null pointer, -- an ``AutoreleasingUnsafeMutablePointer`` value, which is passed verbatim, or -- an inout expression, whose operand is primitive-copied to a temporary - nonowning buffer. The address of that buffer is passed to the callee, and on - return, the value in the buffer is loaded, retained, and reassigned into the - operand. - -Note that the above list does not include arrays, since implicit autoreleasing-to-strong writeback of an entire array would probably not be desirable. - -So if you have a function declared:: - - func bas(x: AutoreleasingUnsafeMutablePointer) - -You can call it as any of:: - - var x: NSBas? = nil - var p: AutoreleasingUnsafeMutablePointer = nil - bas(nil) - bas(p) - bas(&x) - -UnsafePointer ---------------------- - -When a function is declared as taking an ``UnsafeMutablePointer`` argument, it can -accept any of the following: - -- nil, which is passed as a null pointer, -- an ``UnsafeMutablePointer``, ``UnsafePointer``, or - ``AutoreleasingUnsafeMutablePointer`` value, which is converted to - ``UnsafePointer`` if necessary and passed verbatim, -- an inout expression whose operand is an lvalue of type ``T``, which is passed - as the address of (the potentially temporary writeback buffer of) the lvalue, - or -- an ``Array`` value, which is passed as a pointer to the start of the - array, and lifetime-extended for the duration of the callee. - -As a special case, when a function is declared as taking an -``UnsafePointer`` argument, it can accept the same operands as -``UnsafePointer`` for any type ``T``. Pointers to certain integer -types can furthermore interoperate with strings; see `Strings`_ below. - -So if you have a function declared:: - - func zim(x: UnsafePointer) - -You can call it as any of:: - - var x: Float = 0.0 - var p: UnsafePointer = nil - zim(nil) - zim(p) - zim(&x) - zim([1.0, 2.0, 3.0]) - -And if you have a function declared:: - - func zang(x: UnsafePointer) - -You can call it as any of:: - - var x: Float = 0.0, y: Int = 0 - var p: UnsafePointer = nil, q: UnsafePointer = nil - zang(nil) - zang(p) - zang(q) - zang(&x) - zang(&y) - let doubles = [1.0, 2.0, 3.0] - let ints = [1, 2, 3] - zang(doubles) - zang(ints) - -A type checker limitation prevents array literals from being passed directly -to ``UnsafePointer`` arguments without type annotation. As a -workaround, you can bind the array literal to a constant, as above, or -specify the array type with ``as``:: - - zang([1.0, 2.0, 3.0] as [Double]) - zang([1, 2, 3] as [Int]) - -This limitation is tracked as . - -Strings -======= - -Pointers to the following C integer and character types can interoperate with -Swift ``String`` values and string literals: - -- ``CChar``, ``CSignedChar``, and ``CUnsignedChar``, which interoperate with - ``String`` as a UTF-8 code unit array; -- (not implemented yet) ``CShort``, ``CUnsignedShort``, and ``CChar16``, which interoperate with - ``String`` as a UTF-16 code unit array; and -- (not implemented yet) ``CInt``, ``CUnsignedInt``, ``CWideChar``, and ``CChar32``, which interoperate - with ``String`` as a UTF-32 code unit array. - -A ``UnsafePointer`` parameter with any of the above element types may take -a ``String`` value as an argument. The string is transcoded to a null-terminated -buffer of the appropriate encoding, if necessary, and a pointer to the buffer -is passed to the function. The callee may not mutate through the array, and -the referenced memory is only guaranteed to live for the duration of the call. - diff --git a/docs/proposals/Concurrency.md b/docs/proposals/Concurrency.md new file mode 100644 index 0000000000000..32e9c3881a116 --- /dev/null +++ b/docs/proposals/Concurrency.md @@ -0,0 +1,701 @@ +Swift Thread Safety +=================== + +This document describes the Swift thread-safety layer. It includes the +motivation for allowing users to write thread-safe code and a concrete +proposal for changes in the language and the standard library. This is a +proposal and not a plan of record. + +The low-level thread-safety layer allows third-party developers to build +different kinds of high-level safe concurrency solutions (such as +Actors, Coroutines, and Async-Await) in a library. This document +describes three different high-level concurrency solutions to +demonstrate the completeness and efficacy of the thread-safe layer. +Designing an official high-level concurrency model for Swift is outside +the scope of this proposal. + +Motivation and Requirements +--------------------------- + +Multi-core processors are ubiquitous and most modern programming +languages, such as Go, Rust, Java, C\#, D, Erlang, and C++, have some +kind of support for concurrent, parallel or multi-threaded programming. +Swift is a safe programming language that protects the user from bugs +such as integer overflow and memory corruption by eliminating undefined +behavior and by verifying some aspects of the program's correctness at +runtime. Multi-threaded programs in Swift should not break the safety +guarantees of the language. + +Swift Memory Model +------------------ + +This section describes the guarantees that unsupervised Swift provides +when writing multi-threaded code and why the minimal guarantees of +atomicity that Java provide cannot be implemented in Swift. + +Let's start by looking at reference counting operations. Swift objects +include a reference-count field that holds the number of references that +point to that object. This field is modified using atomic operations. +Atomic operations ensure that the reference count field is not corrupted +by data races. However, using atomic operations is not enough to ensure +thread safety. Consider the multi-threaded program below. + +``` {.sourceCode .swift} +import Foundation + +let priority = DISPATCH_QUEUE_PRIORITY_DEFAULT +let queue = dispatch_get_global_queue(priority, 0) + +class Bird {} +var single = Bird() + +dispatch_async(queue) { + while true { single = Bird() } +} +while true { single = Bird() } +``` + +This program crashes very quickly when it tries to deallocate an already +deallocated class instance. To understand the bug try to imagine two +threads executing the SIL code below in lockstep. After they both load +the same value they both try to release the object. One thread succeeds +and deallocates the object while another thread attempts to read the +memory of a deallocated object: + + %10 = global_addr @singleton : $*Bird + + bb: + %49 = alloc_ref $Bird + %51 = load %10 : $*Bird + store %49 to %10 : $*Bird + strong_release %51 : $Bird + br bb + +Next, we'll look into the problem of sliced values. Intuitively, it is +easy to see why sharing memory between two threads could lead to +catastrophic bugs. Consider the program below: + + Thread #1: Thread #2: + A.first = "John" A.first = "Paul" + A.last = "Lennon" A.last = "McCartney" + +If thread \#1 goes to sleep after executing the first statement and +resumes execution after thread \#2 runs, then the value of the struct +would be "Paul Lennon", which is not a valid value. The example above +demonstrates how data races can introduce an inconsistent state. + +The Java memory model ensures that pointers and primitive types, such as +ints and floats, are never sliced, even when data races occur. It would +be nice if Swift had a similar guarantee. Intuitively we would want all +struct or class members of primitive types to be aligned to ensure +atomic access to the field. However, this is not possible in Swift. +Consider the Swift code below: + + enum Fruit { + case Apple(Int64), + case Grape(MyClass) + } + +The size of the struct above is 65 bits, which means that even on 64bit +processors the tag of the enum can't be updated at the same time as the +payload. In some race conditions we could accidentally interpret the +payload of the struct as pointer using the value stored into the +integer. + +To summarize, Swift, just like C++, does not make any guarantees about +unsynchronized memory, and the semantic of programs with races is +undefined. When race conditions occur pointers and primitive data types +could be sliced, enums may contain the wrong tag, protocols may refer to +invalid dispatch tables, references may point to deallocated objects. + +Achieving thread safety +----------------------- + +This section describes a set of rules that ensure thread safety in +programs that embrace them despite the inherit lack of thread safety in +general multi-threaded Swift code. + +Safe concurrency is commonly implemented by eliminating shared mutable +memory. Go, Erlang and Rust ensure some level of program safety by +providing mechanisms for eliminating shared mutable memory. Erlang +provides the strongest model by ensuring complete logical address space +separation between threads. Rust provides powerful abstraction and rely +on the type system to ensure that objects are owned by a single entity. +Go provides channels that allow threads to communicate instead of +sharing memory (but allows user to pass pointers in channels!). It is +not necessary to disallow all sharing of mutable data between threads +and it is not necessary to enforce a hermetic separation between the +address spaces. It is very useful to be able to share large data +structures without copying them around. Mutable data can be shared +between threads as long as the access to the data is synchronized and +some program properties are verified by the compiler. In Swift thread +safety is implemented by preventing threads from sharing mutable memory. + +Proposal +======== + +In Swift, new threads are created in a new memory enclave that is +separate from the parent thread. Values can be copied in and out of the +new thread context, but the child thread must never obtain a reference +that points to the outside world. Non-reentrant code needs to be +explicitly marked as such. Swift enforces these rules statically. The +rest of this section describes how Swift ensures safety and deals with +global variables and unsafe code. + +The three basic elements of thread safety +----------------------------------------- + +The Swift language has three features that allow it to ensure thread +safety and enforce it at compile time: + +### 1. Copyable Protocol + +The **Copyable protocol** marks types of instances that can be copied +from one thread context to another. + +Instances of some types, such as Int, can be copied safely between +threads because they do not contain references that allow threads to +access memory that they do not own. Some types, such as String and Array +(with copyable elements) can be copied between thread context because +they have value semantics and the internal reference is not exposed. + +The compiler derives the conformance of POD types and trivial enums to +the Copyable protocol automatically. Library designers need to manually +mark types with value semantics as Copyable. + +Value-semantic types are not the only category of types that can be +copied. Library designers can implement thread-safe or lockless data +structures and manually mark them as Copyable. + +Notice that due to rdar://17144340 we still can't mark Arrays and +Optionals as copyable: + + // Optionals are copyable if the payload type is copyable. + extension Optional : CopyableType where T : CopyableType {} + +### 2. Reentrant code + +We ensure thread-safety by requiring that code that's executed from a +worker thread to only access logical copies of data that belongs to +other threads. One way for user code to break away from the memory +enclave is to access **global variables**. The Swift compiler must +verify that threaded code does not access global variables or unsafe +code that it can't verify. There are exceptions to this rule and the +compiler provides special annotations for code that performs I/O or +calls unsafe code. + +**Reentrant** code is code that only accesses memory that is accessible +from the passed arguments. In other words, reentrant code does not +access global variables or shared resources. + +The thread verifier needs to be able to analyze all of the code that +could potentially be executed by a work thread and ensure that it is +reentrant. Dynamically dispatched calls, file and module boundary limit +the efficacy of the thread-verifier. This means that the information of +whether a function is reentrant or not needs to be a part of the +**function signature**. + +The **unsafe** attribute is used to denote code that is allowed to +access global variables and unsafe code. Objective-C methods are +automatically marked as **'unsafe'** unless they are explicitly marked +with the **safe** attribute. The safe and unsafe attributes provide a +migration path for large bodies of code that do not explicitly mark the +APIs as reentrant or non-reentrant. + +In the example program below the method fly may access the global +variable because it is marked with the attribute unsafe. The compile +won't allow this method to be executed from a worker-thread. + +``` {.sourceCode .swift} +var glob : Int = 1 + +class Bird { + unsafe func fly() { glob = 1} +} +``` + +In the example program below the issafe wrapper is used to explicitly +mark a region as safe. The developer is pacifying the compiler and +explicitly marking the code as safe. + +The function `logger` is still considered by the compiler as reentrant +and can be called by worker-threads. + +``` {.sourceCode .swift} +func logger(x : Int) { + + // I know what I'm doing! + issafe { + glob = x + } +} +``` + +Most protocols in the standard library, like Incrementable and Equatable +are annotated as safe by default. + +### 3. Gateways annotation + +Gateway annotation is a special semantics annotation that marks +functions that create new threads. This allows the compiler to verify +that all of the arguments that are passed to the thread conform to the +Copyable protocol and that the code that is executed by the worker +thread is reentrant. + +The compiler also verifies a few requirements that are special to the +thread creation site, like making sure that the closure to be executed +does not capture local mutable variables. + +Library developers who implement high-level concurrency libraries can +use the gateway annotation to mark the functions that launch new +threads. + +``` {.sourceCode .swift} +@_semantics("swift.concurrent.launch") +public func createTask(args : ArgsTy, callback : (ArgsTy) -> Void) { + ... +} +``` + +Summary +------- + +Together, the thread verifier, the Copyable protocol, and the gateway +annotation allow us to implement the thread-safety layer. The rest of +this document demonstrates how these features are used for the +implementation of high-level concurrency systems. + +The implementations of the thread-safety layer, the thread verifier, and +programs that use the three concurrency libraries are available in the +`concurrency` git branch. + +Implementing safe Go-lang style concurrency +=========================================== + +In this section, we describe how the proposed thread-safety layer can be +used for implementing go-lang style concurrency. Go supports concurrency +using coroutines and channels. We are going to demonstrate how to +implement go-style concurrency using verified code, CopyableType +protocol and gateway annotations. + +Let's start by implementing Streams, which are analogous to go channels. +A stream is simply a blocking queue with restrictions on the types that +can be passed. Streams are generic data structures where the queue +element type is `CopyableType` (and conforms to the relevant protocol, +discussed above). Streams are the only legitimate channel of +communication between threads. + +Streams can be shared by multiple tasks. These tasks can read from and +write into the stream concurrently. Reads from streams that contain no +data and writes into full streams will be blocked, meaning that the +operating system will put the calling thread to sleep and wait for new +data to arrive to wake the sleeping thread. This property allows the +Stream to be used as a synchronization mechanism. + +The second half of the go concurrency feature is coroutines. In Swift +lingo, we'll call them Tasks. Tasks are functions that are executed by +threads asynchronously. Tasks could have their own stack (this is an +implementation detail that is not important at this point) and can run +indefinitely. Tasks are created using gateways (see above) that ensure +thread safety. + +Together tasks and streams create a thread-safe concurrency construct. +Let's delve into this claim. Tasks are created using gateways that +ensure that all arguments being passed into the closure that will be +executed are CopyableType. In other words, all of the arguments are +either deep-copied or implemented in a way that will forbid sharing of +memory. The gateway also ensures that the closure that will be executed +by the task is verified, which means that it will not access global +variables or unsafe code, and it will not capture any variable that is +accessible by the code that is creating the task. This ensures a +hermetic separation between the newly created thread and the parent +thread. Tasks can communicate using streams that ensure that information +that passes between threads, just like the task's closure arguments, +does not leak references and keeps the hermetic separation between the +tasks. Notice that Streams themselves are CopyableTypes because they can +be copied freely between tasks without violating thread safety. + +Stream and Tasks provide safety and allow users to develop server-like +tasks easily. Reading requests from a queue, processing the request and +writing it into another queue are easy, especially since the queues +themselves provide the synchronization mechanisms. Deadlocks manifest +themselves as read requests from an empty queue, which makes debugging +and reasoning about these bugs trivial. + +Usage Example +------------- + +This is an example of a tiny concurrent program that uses Tasks and +Streams. + +``` {.sourceCode .swift} +let input = Stream() +let output = Stream() + +func echoServer(inp : Stream, + out : Stream) { + while true { out.push(inp.pop()) } +} + +createTask((input, output), callback: echoServer) + +for val in ["hello","world"] { + input.push(val) + print(output.pop()) +} +``` + +The program above creates a server task that accepts an input stream and +an output stream that allows it to communicate with the main thread. The +compiler verifies that the task does not access any disallowed memory +locations (as described below). + +It is entirely possible to remove the manual declaration of the streams +and the argument types and define a single endpoint for communication +with the new task. In the example below the type declaration of the +endpoint helps the type checker to deduct the type of the stream +arguments and allows the developer to omit the declaration of the +streams in the closure. + +``` {.sourceCode .swift} +let comm : _Endpoint = createTask({var counter = 0 + while true { + $0.pop() + $0.push(counter++) + }}) +// CHECK: 0, 1, 2, +for ss in ["","",""] { + comm.push(ss) + print("\(comm.pop()), ", appendNewline: false) +} +``` + +Stream utilities +---------------- + +The Swift library can to implement a few utilities that will allow users +and library designers to build cool things: + +- The `Funnel` class accepts multiple incoming streams and weaves them + into a single outgoing stream. +- The `Fan-out` class accepts a single incoming stream and duplicates + the messages into multiple outgoing streams. +- The `waitForStream` function accepts multiple Streams and returns + only when one or more of the streams are ready to be read. + +It is entirely possible to implement MPI-like programs that broadcast +messages or send messages to a specific task. It is also very easy to +implement barriers for SPMD-like programs using fan-out stream. + +Implementing Async - Await +========================== + +Async-Await is one of the most popular and effective concurrency +solutions. In this section we describe how the proposed thread-safety +layer can be used for implementing Async-Await style concurrency. + +Async calls are function calls that return a Future, which is a +mechanism that allows the caller of asynchronous procedures to wait for +the results. The async call execute the callback closure in a secure +enclave to ensure thread safety. + +Example +------- + +Example of a concurrent program using Futures in Swift. + +``` {.sourceCode .swift} +func merge_sort(array: ArraySlice) -> [T] { + + if array.count <= 16 { return Array(array).sort() } + + let mid = array.count / 2 + let left = array[0.. (String) in + switch c { + case .Circle: return "Circle" + case .Oval: return "Oval" + case .Square: return "Square" + case .Triangle: return "Triangle" + }} + +//CHECK: Shape: Oval +print("Shape: \( res.await() )") +``` + +Notice that the swift compiler infers that `Shape` and String can be +sent between the threads. + +UI programming with Async +------------------------- + +One of the goals of this proposal is to allow users to develop +multi-threaded UI applications that are safe. + +At the moment Swift users that use GCD are advised to start a new block +in a new thread. Once the task finishes the recommendation is to +schedule another block that will be executed by the main event loop. + +Notice that the Async call returns a Future, and the callee needs to +block on the result of the Future. In this section we describe the +extension to the Async call that allows it to execute code on the main +event loop asynchronously. + +One possible solution would be to add an async call that accepts two +closures. One that's executed asynchronously, and another one that will +be executed synchronously after the task is finished. F\# provides a +similar API (with StartWithContinuations). + +One possible implementation is one where the task creation call return +an object that allows the users to register callbacks of different +kinds. The destructor of the task object would execute the work callback +for convenience. The two useful callbacks are "on completion" that would +execute code in the main UI thread and "on error" that would be executed +in case of an exception in the work closure. + +This is a small example from an app that counts the number of prime +numbers between one and million concurrently. The first closure is the +worker closure that does all the work in a separate thread (and is +verified by the thread safety checker), and the second closure is +executed by the UI main loop and is free to make unsafe calls capture +locals and access globals. + +``` {.sourceCode .swift} +@IBAction func onClick(sender: AnyObject) { + + progress.startAnimating() + Label!.text = "" + + asyncWith (1_000_000) { + (num: Int) -> Int in + var sum = 0 + for i in 1..(args : ArgsTy, callback : (ArgsTy) -> RetTy) -> Future { + return unsafeAsync(args, callback: callback) +} +``` + +Example of shared data structures +--------------------------------- + +In the example below the class PrimesCache is explicitly marked by the +user as a CopyableType. The user implemented a thread-safe class that +allows concurrent access to the method `isPrime`. To implement a +critical section the user inherit the class `Sync` that contains a lock +and a method that implements a critical section. The user also had to +annotate the shared method as safe because the verifier has no way of +knowing if the call is safe. Notice that the critical section itself is +not enough to ensure thread safety because the critical section could be +accessing memory that is shared between threads that are not +synchronized on the same lock. + +``` {.sourceCode .swift} +final class PrimesCache : Sync, CopyableType { + var cache : [Int : Bool] = [:] + + @_semantics("swift.concurrent.safe") + func isPrime(num : Int) -> Bool { + return self.critical { + if let r = self.cache[num] { return r } + let b = calcIsPrime(num) + self.cache[num] = b + return b + } + } +} + +func countPrimes(P : PrimesCache) -> Int { + var sum = 0 + for i in 2..<10_000 { if P.isPrime(i) { sum += 1} } + return sum +} + +let shared = PrimesCache() +let R1 = async(shared, callback: countPrimes) +let R2 = async(shared, callback: countPrimes) + +// CHECK: [1229, 1229] +print([R1.await(), R2.await()]) +``` + +Example of parallel matrix multiply using Async +----------------------------------------------- + +This is a small example of the parallel matrix multiplication algorithm +using async and futures. The slices of the matrix are not copied when +they are moved between the threads because ContiguousArray has value +semantics and the parallel code runs significantly faster. + +``` {.sourceCode .swift} +func ParallelMatMul(A : Matrix,_ B : Matrix) -> Matrix { + assert(A.size == B.size, "size mismatch!") + + // Handle small matrices using the serial algorithm. + if A.size < 65 { return SerialMatMul(A, B) } + + var product = Matrix(A.size) + // Extract 4 quarters from matrices A and B. + let half = A.size/2 + let A11 = A.slice(half ,0 , 0) + let A12 = A.slice(half ,0 , half) + let A21 = A.slice(half ,half, 0) + let A22 = A.slice(half ,half, half) + let B11 = B.slice(half ,0 , 0) + let B12 = B.slice(half ,0 , half) + let B21 = B.slice(half ,half, 0) + let B22 = B.slice(half ,half, half) + + // Multiply each of the sub blocks. + let C11_1 = async((A11, B11), callback: ParallelMatMul) + let C11_2 = async((A12, B21), callback: ParallelMatMul) + let C12_1 = async((A11, B12), callback: ParallelMatMul) + let C12_2 = async((A12, B22), callback: ParallelMatMul) + let C21_1 = async((A21, B11), callback: ParallelMatMul) + let C21_2 = async((A22, B21), callback: ParallelMatMul) + let C22_1 = async((A21, B12), callback: ParallelMatMul) + let C22_2 = async((A22, B22), callback: ParallelMatMul) + + // Add the matching blocks. + let C11 = C11_1.await() + C11_2.await() + let C12 = C12_1.await() + C12_2.await() + let C21 = C21_1.await() + C21_2.await() + let C22 = C22_1.await() + C22_2.await() + + // Save the matrix slices into the correct locations. + product.update(C11, 0 , 0) + product.update(C12, 0 , half) + product.update(C21, half, 0) + product.update(C22, half, half) + return product +} +``` + +Implementing Actors +=================== + +In this section we describe how the proposed thread-safety layer can be +used for implementing Actor-based concurrency. + +Actors communicate using asynchronous messages that don't block. Systems +that use actors can scale to support millions of concurrent actors +because actors are not backed by a live thread or by a stack. + +In Swift actors could be implemented using classes that inherit from the +generic `Actor` class. The generic parameter determines the type of +messages that the actor can accept. The message type needs to be of +`CopyableType` to ensure the safety of the model. The actor class +exposes two methods: `send` and `accept`. Messages are sent to actors +using the `send` method and they never block the sender. Actors process +the message using the `accept` method. + +At this point it should be obvious to the reader of the document why +marking the `accept` method as thread safe and allowing the parameter +type to be `CopyableType` will ensure the safety of the system (this is +discussed at length in the previous sections). + +The `accept` method is executed by a user-space scheduler and not by +live thread and this allows the system to scale to tens of thousands of +active actors. + +The code below depicts the famous prime numbers sieve program using +actors. The sieve is made of a long chain of actors that pass messages +to one another. Finally, a collector actor saves all of the messages +into an array. + +``` {.sourceCode .swift} +// Simply collect incoming numbers. +class Collector : Actor { + + var numbers = ContiguousArray() + + override func accept(x : Int) { numbers.append(x) } +} + +// Filter numbers that are divisible by an argument. +class Sieve : Actor { + var div : Int + var next : Actor + + init(div d : Int, next n : Actor) { + div = d ; next = n + } + + override func accept(x : Int) { + if x != div && x % div == 0 { return } + next.send(x) + } +} + +var col = Collector() +var head : Actor = col + +// Construct the Sieve +for i in 2..(args : ArgsTy, callback : (ArgsTy) -> Void) { - ... - } - - -Summary -------- - -Together, the thread verifier, the Copyable protocol, and the gateway annotation -allow us to implement the thread-safety layer. The rest of this document demonstrates -how these features are used for the implementation of high-level -concurrency systems. - -The implementations of the thread-safety layer, the thread verifier, and -programs that use the three concurrency libraries are available in the -``concurrency`` git branch. - -Implementing safe Go-lang style concurrency -=========================================== - -In this section, we describe how the proposed thread-safety layer can be used for -implementing go-lang style concurrency. Go supports concurrency using -coroutines and channels. We are going to demonstrate how to -implement go-style concurrency using verified code, CopyableType protocol -and gateway annotations. - -Let's start by implementing Streams, which are analogous to go channels. A -stream is simply a blocking queue with restrictions on the types that can be -passed. Streams are generic data structures where the queue element type is -``CopyableType`` (and conforms to the relevant protocol, discussed above). -Streams are the only legitimate channel of communication between threads. - -Streams can be shared by multiple tasks. These tasks can read from and write into the stream -concurrently. Reads from streams that contain no data and writes into full streams -will be blocked, meaning that the operating system will put the calling thread to sleep and wait for -new data to arrive to wake the sleeping thread. -This property allows the Stream to be used as a synchronization mechanism. - -The second half of the go concurrency feature is coroutines. In Swift lingo, -we'll call them Tasks. Tasks are functions that are executed by threads -asynchronously. Tasks could have their own stack (this is an implementation -detail that is not important at this point) and can run indefinitely. Tasks are -created using gateways (see above) that ensure thread safety. - -Together tasks and streams create a thread-safe concurrency construct. Let's -delve into this claim. Tasks are created using gateways that ensure that all -arguments being passed into the closure that will be executed are -CopyableType. In other words, all of the arguments are either deep-copied or -implemented in a way that will forbid sharing of memory. The gateway also -ensures that the closure that will be executed by the task is verified, which -means that it will not access global variables or unsafe code, and it will not capture -any variable that is accessible by the code that is creating the task. This -ensures a hermetic separation between the newly created thread and the parent -thread. Tasks can communicate using streams that ensure that information that -passes between threads, just like the task's closure arguments, does not leak -references and keeps the hermetic separation between the tasks. Notice that -Streams themselves are CopyableTypes because they can be copied freely between -tasks without violating thread safety. - -Stream and Tasks provide safety and allow users to develop server-like tasks -easily. Reading requests from a queue, processing the request and writing it into -another queue are easy, especially since the queues themselves provide the -synchronization mechanisms. Deadlocks manifest themselves as read requests from -an empty queue, which makes debugging and reasoning about these bugs trivial. - -Usage Example -------------- -This is an example of a tiny concurrent program that uses Tasks and Streams. - -.. code-block:: swift - - let input = Stream() - let output = Stream() - - func echoServer(inp : Stream, - out : Stream) { - while true { out.push(inp.pop()) } - } - - createTask((input, output), callback: echoServer) - - for val in ["hello","world"] { - input.push(val) - print(output.pop()) - } - -The program above creates a server task that accepts an input stream and an -output stream that allows it to communicate with the main thread. The compiler -verifies that the task does not access any disallowed memory locations (as -described below). - -It is entirely possible to remove the manual declaration of the streams and the -argument types and define a single endpoint for communication with the new task. -In the example below the type declaration of the endpoint helps the type checker -to deduct the type of the stream arguments and allows the developer to omit the -declaration of the streams in the closure. - -.. code-block:: swift - - let comm : _Endpoint = createTask({var counter = 0 - while true { - $0.pop() - $0.push(counter++) - }}) - // CHECK: 0, 1, 2, - for ss in ["","",""] { - comm.push(ss) - print("\(comm.pop()), ", appendNewline: false) - } - -Stream utilities ----------------- -The Swift library can to implement a few utilities that will allow users and -library designers to build cool things: - -* The ``Funnel`` class accepts multiple incoming streams and weaves them into a - single outgoing stream. - -* The ``Fan-out`` class accepts a single incoming stream and duplicates the - messages into multiple outgoing streams. - -* The ``waitForStream`` function accepts multiple Streams and returns only when - one or more of the streams are ready to be read. - -It is entirely possible to implement MPI-like programs that broadcast messages -or send messages to a specific task. It is also very easy to implement barriers -for SPMD-like programs using fan-out stream. - - -Implementing Async - Await -========================== - -Async-Await is one of the most popular and effective concurrency solutions. In -this section we describe how the proposed thread-safety layer can be used for -implementing Async-Await style concurrency. - -Async calls are function calls that return a Future, which is a mechanism that -allows the caller of asynchronous procedures to wait for the results. The async -call execute the callback closure in a secure enclave to ensure thread safety. - -Example -------- -Example of a concurrent program using Futures in Swift. - -.. code-block:: swift - - func merge_sort(array: ArraySlice) -> [T] { - - if array.count <= 16 { return Array(array).sort() } - - let mid = array.count / 2 - let left = array[0.. (String) in - switch c { - case .Circle: return "Circle" - case .Oval: return "Oval" - case .Square: return "Square" - case .Triangle: return "Triangle" - }} - - //CHECK: Shape: Oval - print("Shape: \( res.await() )") - -Notice that the swift compiler infers that ``Shape`` and `String` can be sent -between the threads. - -UI programming with Async -------------------------- - -One of the goals of this proposal is to allow users to develop multi-threaded UI -applications that are safe. - -At the moment Swift users that use GCD are advised to start a new block in a new -thread. Once the task finishes the recommendation is to schedule another block -that will be executed by the main event loop. - -Notice that the Async call returns a Future, and the callee needs to block on -the result of the Future. In this section we describe the extension to the -Async call that allows it to execute code on the main event loop asynchronously. - -One possible solution would be to add an async call that accepts two closures. -One that's executed asynchronously, and another one that will be executed -synchronously after the task is finished. F# provides a similar API (with -StartWithContinuations). - -One possible implementation is one where the task creation call return an object -that allows the users to register callbacks of different kinds. The destructor -of the task object would execute the work callback for convenience. The two -useful callbacks are "on completion" that would execute code in the main UI -thread and "on error" that would be executed in case of an exception in the work -closure. - -This is a small example from an app that counts the number of prime numbers -between one and million concurrently. The first closure is the worker closure -that does all the work in a separate thread (and is verified by the thread -safety checker), and the second closure is executed by the UI main loop and is -free to make unsafe calls capture locals and access globals. - -.. code-block:: swift - - @IBAction func onClick(sender: AnyObject) { - - progress.startAnimating() - Label!.text = "" - - asyncWith (1_000_000) { - (num: Int) -> Int in - var sum = 0 - for i in 1..(args : ArgsTy, callback : (ArgsTy) -> RetTy) -> Future { - return unsafeAsync(args, callback: callback) - } - -Example of shared data structures ---------------------------------- - -In the example below the class PrimesCache is explicitly marked by the user as a -CopyableType. The user implemented a thread-safe class that allows concurrent -access to the method ``isPrime``. To implement a critical section the user -inherit the class ``Sync`` that contains a lock and a method that implements a -critical section. The user also had to annotate the shared method as safe -because the verifier has no way of knowing if the call is safe. Notice that the -critical section itself is not enough to ensure thread safety because the -critical section could be accessing memory that is shared between threads that -are not synchronized on the same lock. - -.. code-block:: swift - - final class PrimesCache : Sync, CopyableType { - var cache : [Int : Bool] = [:] - - @_semantics("swift.concurrent.safe") - func isPrime(num : Int) -> Bool { - return self.critical { - if let r = self.cache[num] { return r } - let b = calcIsPrime(num) - self.cache[num] = b - return b - } - } - } - - func countPrimes(P : PrimesCache) -> Int { - var sum = 0 - for i in 2..<10_000 { if P.isPrime(i) { sum += 1} } - return sum - } - - let shared = PrimesCache() - let R1 = async(shared, callback: countPrimes) - let R2 = async(shared, callback: countPrimes) - - // CHECK: [1229, 1229] - print([R1.await(), R2.await()]) - - -Example of parallel matrix multiply using Async ------------------------------------------------ - -This is a small example of the parallel matrix multiplication algorithm using -async and futures. The slices of the matrix are not copied when they are moved -between the threads because ContiguousArray has value semantics and the parallel -code runs significantly faster. - -.. code-block:: swift - - func ParallelMatMul(A : Matrix,_ B : Matrix) -> Matrix { - assert(A.size == B.size, "size mismatch!") - - // Handle small matrices using the serial algorithm. - if A.size < 65 { return SerialMatMul(A, B) } - - var product = Matrix(A.size) - // Extract 4 quarters from matrices A and B. - let half = A.size/2 - let A11 = A.slice(half ,0 , 0) - let A12 = A.slice(half ,0 , half) - let A21 = A.slice(half ,half, 0) - let A22 = A.slice(half ,half, half) - let B11 = B.slice(half ,0 , 0) - let B12 = B.slice(half ,0 , half) - let B21 = B.slice(half ,half, 0) - let B22 = B.slice(half ,half, half) - - // Multiply each of the sub blocks. - let C11_1 = async((A11, B11), callback: ParallelMatMul) - let C11_2 = async((A12, B21), callback: ParallelMatMul) - let C12_1 = async((A11, B12), callback: ParallelMatMul) - let C12_2 = async((A12, B22), callback: ParallelMatMul) - let C21_1 = async((A21, B11), callback: ParallelMatMul) - let C21_2 = async((A22, B21), callback: ParallelMatMul) - let C22_1 = async((A21, B12), callback: ParallelMatMul) - let C22_2 = async((A22, B22), callback: ParallelMatMul) - - // Add the matching blocks. - let C11 = C11_1.await() + C11_2.await() - let C12 = C12_1.await() + C12_2.await() - let C21 = C21_1.await() + C21_2.await() - let C22 = C22_1.await() + C22_2.await() - - // Save the matrix slices into the correct locations. - product.update(C11, 0 , 0) - product.update(C12, 0 , half) - product.update(C21, half, 0) - product.update(C22, half, half) - return product - } - - -Implementing Actors -=================== - -In this section we describe how the proposed thread-safety layer can be used for -implementing Actor-based concurrency. - -Actors communicate using asynchronous messages that don't block. Systems that -use actors can scale to support millions of concurrent actors because actors are -not backed by a live thread or by a stack. - -In Swift actors could be implemented using classes that inherit from the generic -``Actor`` class. The generic parameter determines the type of messages that the -actor can accept. The message type needs to be of ``CopyableType`` to ensure the -safety of the model. The actor class exposes two methods: ``send`` and -``accept``. Messages are sent to actors using the ``send`` method and they never -block the sender. Actors process the message using the ``accept`` method. - -At this point it should be obvious to the reader of the document why -marking the ``accept`` method as thread safe and allowing the parameter type to -be ``CopyableType`` will ensure the safety of the system (this is discussed at -length in the previous sections). - -The ``accept`` method is executed by a user-space scheduler and not by live -thread and this allows the system to scale to tens of thousands of active -actors. - -The code below depicts the famous prime numbers sieve program using actors. The -sieve is made of a long chain of actors that pass messages to one another. -Finally, a collector actor saves all of the messages into an array. - -.. code-block:: swift - - // Simply collect incoming numbers. - class Collector : Actor { - - var numbers = ContiguousArray() - - override func accept(x : Int) { numbers.append(x) } - } - - // Filter numbers that are divisible by an argument. - class Sieve : Actor { - var div : Int - var next : Actor - - init(div d : Int, next n : Actor) { - div = d ; next = n - } - - override func accept(x : Int) { - if x != div && x % div == 0 { return } - next.send(x) - } - } - - var col = Collector() - var head : Actor = col - - // Construct the Sieve - for i in 2.. Element? + } + + struct IntRangeGenerator : SequenceType { + var current: Int + let limit: Int + + // infers SequenceType's Element == Int + mutating func generate() -> Int? { + if current == limit { return nil } + return current++ + } + } + +Type witness inference is a global problem, which involves (among other +things) matching the requirements of a protocol to potential witnesses +within the model type as well as protocol extensions, performing +overload resolution to find the best potential witness, and validating +that the potential type witnesses meets the requirements of the +protocol. Supporting this feature correctly likely means recording type +variables in the AST for type witnesses that are being inferred. + +**Inferring a property's type from its initial value**: The type of a +property can be inferred from its initial value, which makes the +declaration type checker dependent on the expression type checker. This +requires recursion checking that goes through the expression type +checker to diagnose, e.g.: + + var x = y + z + var y = 1 + var z = x + y + +Fortunately, this should be fairly simple: when inferring the type of a +property, it can be temporarily recorded as having unresolved type. Any +attempt to refer to a property of unresolved type within the expression +type checker will be considered ill-formed due to recursion. + +Proposed Architecture +--------------------- + +To address the problems with the current declaration type checker, we +propose a new architecture. The key components of the new architecture +are: + +**Represent phases in the AST**: Each AST node should know to which +phase it has been type-checked. Any accessor on the AST has a +corresponding minimum phase, which it can assert. For example, the +accessor that retrieves the superclass of a class declaration will +assert that the class is at least at the "type hierarchies" phase; it's +programmer error to not have established that the class is at that phase +before asking the question. + +**Model dependencies for phase transitions**: For a given AST node and +target phase, we need to be able to enumerate the phase transitions that +are required of other AST nodes before the transition can be performed. +For example: + + protocol P { + typealias Assoc + } + + struct X : P { + + } + + func foo(x: X.Assoc) { } + +To bring the `TypeRepr` for `X.Assoc` to the "primary name binding" +phase, we need to bring `X` up to the "primary name binding" phase. Once +all dependencies for a phase transition have been resolved, we can +perform the phase transition. As noted earlier, it's important to make +the dependencies minimal: for example, note that we do not introduce any +dependencies on the type argument (`Int`) because it does not affect +name lookup. It could, however, affect declaration type validation. + +**Iteratively solve type checking problems**: Immediately recursing to +satisfy a dependency (as the current type checker does) leads to +unbridged recursion in the type checker. Instead, unsatisfied +dependencies should be pushed into a dependency graph that tracks all of +the active AST node dependencies as well as a priority queue that guides +the type checker to the next AST node whose dependencies have been +satisfied. + +**Detect and diagnose recursive dependencies**: When the priority queue +contains only AST nodes that have dependencies that have not yet been +satisfied, we have a circular dependency in the program. We can find the +cycle within the active dependency graph and report it to the user. + +**Separate semantic information from the AST**: Rather than stash all of +the semantic and type information for declarations directly on the AST, +we can keep it in a separate side table. That makes it easier to handle +both global inference problems (where we need to tentatively make +assumptions about type variables) and also, in the longer term, to +perform more incremental compilation where we can throw away semantic +and type information that has been evaluated. + +**Global symbol table**: Name lookup within a type or extension context +typically requires us to bring that type up to the "type hierarchies" +phase. If we were to use a global symbol table that also had information +about members, name lookup could find declarations using that symbol +table, possibly without having to bring the type context past the +"parsing" phase. + +### Case study: conformance lookup table + +The protocol conformance lookup table (in +`lib/AST/ProtocolConformance.cpp`), which answers questions about the +set of protocols that a given class conforms to, has a similar +architecture to what is proposed here. Each nominal type has a +conformance lookup table, which is lazily constructed from the nominal +type, its extensions, and the conformance lookup table of its superclass +(if any). + +Conformance checking is divided into four phases, modeled by +`ConformanceLookupTable::ConformanceStage`: recording of +explicitly-written conformances, handling of inherited conformances, +expanding out implied conformances (due to protocol inheritance), and +resolving ambiguities among different sources of conformances. The phase +of nominal type declaration and each of its extensions are separately +tracked, which allows for new extensions to be lazily introduced. Phase +transitions are handled by a single method +(`ConformanceLookupTable::updateLookupTable`) that recurses to satisfy +dependencies. For example, bringing a class `C` up to the "inherited" +phase requires that its superclass be brought to the "resolved" phase. + +Whenever the conformance lookup table encounters a problem, such as a +conflict between a superclass's protocol conformance and a subclass's +protocol conformance, it records the problem in a diagnostics side-table +and resolves the conflict in a manner that allows other type checking to +continue. The actual diagnosis of the problem occurs only when +performing complete semantics checking of the declaration that owns the +erroneous protocol conformance. + +Note that the conformance lookup table does *not* implement a dependency +graph or priority queue as proposed above. Rather, it performs direct +recursion internally (which is generally not a problem) and through the +current type-validation logic (which requires it to be re-entrant). + +### How do we get there? + +The proposed architecture is significantly different from the current +type checker architecture, so how do we get there from here? There are a +few concrete steps we can take: + +**Make all AST nodes phase-aware**: Introduce a trait that can ask an +arbitrary AST node (`Decl`, `TypeRepr`, `Pattern`, etc.) its current +phase. AST nodes may compute this information on-the-fly or store it, as +appropriate. For example, a `TypeRepr` can generally determine its phase +based on the existing state of the `IdentTypeRepr` nodes it includes. + +**Make name lookup phase-aware**: Name lookup is currently one of the +worst offenders when violating phase ordering. Parameterize name lookup +based on the phase at which it's operating. For example, asking for name +lookup at the "extension binding" phase might not resolve type aliases, +look into superclasses, or look into protocols. + +**Make type resolution phase-aware**: Type resolution effectively brings +a given `TypeRepr` up to the "declaration type +validation`phase in one shot. Parameterize type resolution based on the target phase, and start minimizing the among of work that the type checking does. Use extension binding as a testbed for these more-minimal dependencies. **Dependency graph and priority queue**: Extend the current-phase trait with an operation that enumerates the dependencies that need to be satisfied to bring a given AST node up to a particular phase. Start with`TypeRepr`nodes, and use the dependency graph and priority queue to satisfy all dependencies ahead of time, eliminating direct recursion from the type-resolution code path. Build circular-dependency detection within this test-bed. **Incremental adoption of dependency graph**: Make other AST nodes (`Pattern`,`VarDecl`, etc.) implement the phase-awareness trait, enumerating dependencies and updating their logic to perform minimal updates. Certain entry points that are used for ad hoc recursion (such as`validateDecl\`\`) +can push/pop dependency graph and priority-queue instances, which leaves +the existing ad hoc recursion checking in place but allows isolated +subproblems to use the newer mechanisms. + +**Strengthen accessor assertions**: As ad hoc recursion gets eliminated +from the type checker, strengthen assertions on the various AST nodes to +make sure the AST node has been brought to the appropriate phase. + +### How do we test it? + +**Existing code continues to work**: As we move various parts of the +type checker over to the dependency graph, existing Swift code should +continue to work, since we'll have fallbacks to the existing logic and +the new type checker should be strictly more lazy than the existing type +checker. + +**Order-independence testing**: One of the intended improvements from +this type checker architecture is that we should get more predictable +order-independent behavior. To check this, we can randomly scramble the +order in which we type-check declarations in the primary source file of +a well-formed module and verify that we get the same results. + +**Compiler crashers**: The compiler crashers testsuite tends to contain +a large number of crashes that are effectively due to infinite recursion +in the type checker. We expect that many of these will be resolved. + +### How do we measure progress? + +The proposed change is a major architectural shift, and it's only +complete when we have eliminated all ad hoc recursion from the front +end. There are a few ways in which we can measure progress along the +way: + +**AST nodes that implement the phase-aware trait**: Eventually, all of +our AST nodes will implement the phase-aware trait. The number of AST +nodes that do properly implement that trait (reporting current phase, +enumerating dependencies for a phase transition) and become part of the +dependency graph and priority queue gives an indication of how far we've +gotten. + +**Accessors that check the current phase**: When we're finished, each of +the AST's accessors should assert that the AST node is in the +appropriate phase. The number of such assertions that have been enabled +is an indication of how well the type checker is respecting the +dependencies. + +**Phases of AST nodes in non-primary files**: With the current type +checker, every AST node in a non-primary file that gets touched when +type-checking the primary file will end up being fully validated +(currently, the "attribute checking" phase). As the type checker gets +more lazy, the AST nodes in non-primary files will trend toward earlier +phases. Tracking the number of nodes in non-primary files at each phase +over time will help us establish how lazy the type checker is becoming. diff --git a/docs/proposals/DeclarationTypeChecker.rst b/docs/proposals/DeclarationTypeChecker.rst deleted file mode 100644 index 6195f87e438c4..0000000000000 --- a/docs/proposals/DeclarationTypeChecker.rst +++ /dev/null @@ -1,183 +0,0 @@ -:orphan: - -.. @raise litre.TestsAreMissing - -Declaration Type Checker -======================== - -.. contents:: - -Purpose -------- - -This document describes some of the problems with our current "declaration" type checker, which validates and assigns types to the various declarations in a Swift programs. It analyzes the dependencies within the Swift type system that need to be reflected within the implementation and proposes an alternative architecture for the type checker that eliminates these problems. - -Problems with the Current Approach ----------------------------------- - -The current declaration type checker---in particular, ``validateDecl``, which assigns a type to a given declaration---is the source of a large number of Swift bugs, including crashes on both well-formed and ill-formed code, different behavior depending on the order of declarations within a file or across multiple files, infinite recursion, and broken ASTs. The main issues are: - -**Conceptual phases are tangled together**: We have a vague notion that there are phases within the compiler, e.g., extension binding occurs before name binding, which occurs before type checking. However, the implementations in the compiler don't respect phases: extension binding goes through type validation, which does both name binding and type checking. Name lookup attempts to do type checking so that it can establish whether one declaration shadows another. - -**Unprincipled recursion**: Whenever type checking some particular declaration requires information about another declaration, it recurses to type-check that declaration. There are dozens of these recursion points scattered throughout the compiler, which makes it impossible to reason about the recursion or deal with, e.g., recursion that is too deep for the program stack. - -**Ad hoc recursion breaking**: When we do encounter circular dependencies, we have scattered checks for recursion based on a number of separate bits stashed in the AST: ``BeingTypeChecked``, ``EarlyAttrValidation``, ``ValidatingGenericSignature``, etc. Adding these checks is unprincipled: adding a new check in the wrong place tends to break working code (because the dependency is something the compiler should be able to handle), while missing a check permits infinite recursion to continue. - -**Type checker does too much work**: validating a declaration is all-or-nothing. It includes computing its type, but also checking redeclarations and overrides, as well as numerous other aspects that a user of that declaration might not care about. Aside from the performance impact of checking too much, this can introduce false circularities in type-checking, because the user might only need some very basic information to continue. - -Phases of Type Checking ------------------------ - -Type checking for Swift can be divided into a number of phases, where each phase depends on information from the previous phases. A phase may depend on information from another declaration also computed in that phase; such cases need to manage dependencies carefully to detect and diagnose circular dependencies. The granularity of phases can differ: earlier phases tend to more global in nature, while later phases tend to be at a much finer granularity (per-declaration, per-``TypeRepr``, etc.). Generally speaking, we want to minimize the amount of work the type checker performs by moving a given declaration only up to the phase that's absolutely required to make a decision. - -**Parsing**: Parsing produces untyped ASTs describing the structure of the code. Name lookup can find module-scope names in the current module, but any lookup for nested names or names from other modules is unavailable at this point. This is a global phase, because we need to have parsed at least the top-level declarations of each file to enable name lookup. - -**Import resolution**: Resolve the import declarations within a source file to refer to modules. Name lookup can now find module-scope names in the current module or any imported module. This is a file-scoped phase, because one need only resolve the imports within a given file when we're trying to resolve names within that file. - -**Extension binding**: Associate each extension with the nominal type that it extends. This requires name lookup on the name of the type being extended (e.g., "B" in "extension B { ... }"). Extending nested types introduces dependencies within this phase. For example:: - - struct A { } - extension A.Inner { } - extension A { struct Inner { } } - -Here, the second extension must be bound before the first can be resolved. This phase is global in nature, because we need to bind all of the extensions to a particular nominal type before we can proceed with later phases for that nominal type. - -**Type hierarchies**: Establish superclass, protocol-refinement, and protocol-conformance relationships. Detect cycles in inheritance hierarchies and break them so that later phases don't need to reason about them. At this point, general name lookup into a nominal type is possible. - -**Primary name binding**: Resolve names to refer to declarations. The vast majority of names within declarations can be bound at this point, with the exception of names that are dependent on a type parameter. - -**Generic signature resolution**: Collect the requirements placed on a set of generic type parameters for a given declaration, establishing equivalence classes among type parameters and associated types, and so on. Once complete, names dependent on a type parameter can be resolved to, e.g., a particular associated type. At this point, all names that occur within declarations are bound. - -**Declaration type validation**: Produces a type for each declaration. This will involve resolving typealiases and type witnesses, among other things. - -**Override checking**: Determine whether a given class member overrides a member of a superclass. - -**Attribute checking**: Determine whether a given member has a particular attribute, e.g., ``@objc``, ``@available``, ``dynamic``. This depends on override checking because many attributes are inherited. Note that most attributes are independent, so this can be thought of as a per-attribute phase. - -**Semantics checking**: Check that a particular declaration meets the semantic requirements of the language. This particular phase is only important for the file that is currently being type-checked. In particular, for any file that is not the primary file, this checking is useless work. - -Challenges in the Language --------------------------- - -There are a few aspects of the language that make it challenging to implement the declaration type checker. In some cases, we simply need to be more careful in the implementation, while others may require us to restrict the language. Some specific challenges: - -**Extension binding requiring later phases**: When an extension refers to a typealias, we end up with a dependency through the typealias. For example, consider:: - - struct B { } - typealias C = B.Inner - extension C { } - extension B { struct Inner { } } - -Here, the name lookup used for the first extension needs to resolve the typealias, which depends on the second extension having already been bound. There is a similar dependency on resolving superclasses beforing binding extensions:: - - class X { struct Inner { } } - class Y : X { } - extension Y.Inner { } - -We can address this problem by restricting the language to disallow extensions via typealiases and limit the name lookup used for extensions to not consider anything in the superclass or within protocols. It's also possible that a sufficiently lazy type checker could resolve such dependencies. - -**Type witness inference**: Type witnesses can be inferred from other requirements. For example:: - - protocol SequenceType { - typealias Element - mutating func generate() -> Element? - } - - struct IntRangeGenerator : SequenceType { - var current: Int - let limit: Int - - // infers SequenceType's Element == Int - mutating func generate() -> Int? { - if current == limit { return nil } - return current++ - } - } - -Type witness inference is a global problem, which involves (among other things) matching the requirements of a protocol to potential witnesses within the model type as well as protocol extensions, performing overload resolution to find the best potential witness, and validating that the potential type witnesses meets the requirements of the protocol. Supporting this feature correctly likely means recording type variables in the AST for type witnesses that are being inferred. - -**Inferring a property's type from its initial value**: The type of a property can be inferred from its initial value, which makes the declaration type checker dependent on the expression type checker. This requires recursion checking that goes through the expression type checker to diagnose, e.g.:: - - var x = y + z - var y = 1 - var z = x + y - -Fortunately, this should be fairly simple: when inferring the type of a property, it can be temporarily recorded as having unresolved type. Any attempt to refer to a property of unresolved type within the expression type checker will be considered ill-formed due to recursion. - - -Proposed Architecture ---------------------- - -To address the problems with the current declaration type checker, we propose a new architecture. The key components of the new architecture are: - -**Represent phases in the AST**: Each AST node should know to which phase it has been type-checked. Any accessor on the AST has a corresponding minimum phase, which it can assert. For example, the accessor that retrieves the superclass of a class declaration will assert that the class is at least at the "type hierarchies" phase; it's programmer error to not have established that the class is at that phase before asking the question. - -**Model dependencies for phase transitions**: For a given AST node and target phase, we need to be able to enumerate the phase transitions that are required of other AST nodes before the transition can be performed. For example:: - - protocol P { - typealias Assoc - } - - struct X : P { - - } - - func foo(x: X.Assoc) { } - -To bring the ``TypeRepr`` for ``X.Assoc`` to the "primary name binding" phase, we need to bring ``X`` up to the "primary name binding" phase. Once all dependencies for a phase transition have been resolved, we can perform the phase transition. As noted earlier, it's important to make the dependencies minimal: for example, note that we do not introduce any dependencies on the type argument (``Int``) because it does not affect name lookup. It could, however, affect declaration type validation. - -**Iteratively solve type checking problems**: Immediately recursing to satisfy a dependency (as the current type checker does) leads to unbridged recursion in the type checker. Instead, unsatisfied dependencies should be pushed into a dependency graph that tracks all of the active AST node dependencies as well as a priority queue that guides the type checker to the next AST node whose dependencies have been satisfied. - -**Detect and diagnose recursive dependencies**: When the priority queue contains only AST nodes that have dependencies that have not yet been satisfied, we have a circular dependency in the program. We can find the cycle within the active dependency graph and report it to the user. - -**Separate semantic information from the AST**: Rather than stash all of the semantic and type information for declarations directly on the AST, we can keep it in a separate side table. That makes it easier to handle both global inference problems (where we need to tentatively make assumptions about type variables) and also, in the longer term, to perform more incremental compilation where we can throw away semantic and type information that has been evaluated. - -**Global symbol table**: Name lookup within a type or extension context typically requires us to bring that type up to the "type hierarchies" phase. If we were to use a global symbol table that also had information about members, name lookup could find declarations using that symbol table, possibly without having to bring the type context past the "parsing" phase. - -Case study: conformance lookup table -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The protocol conformance lookup table (in ``lib/AST/ProtocolConformance.cpp``), which answers questions about the set of protocols that a given class conforms to, has a similar architecture to what is proposed here. Each nominal type has a conformance lookup table, which is lazily constructed from the nominal type, its extensions, and the conformance lookup table of its superclass (if any). - -Conformance checking is divided into four phases, modeled by ``ConformanceLookupTable::ConformanceStage``: recording of explicitly-written conformances, handling of inherited conformances, expanding out implied conformances (due to protocol inheritance), and resolving ambiguities among different sources of conformances. The phase of nominal type declaration and each of its extensions are separately tracked, which allows for new extensions to be lazily introduced. Phase transitions are handled by a single method (``ConformanceLookupTable::updateLookupTable``) that recurses to satisfy dependencies. For example, bringing a class ``C`` up to the "inherited" phase requires that its superclass be brought to the "resolved" phase. - -Whenever the conformance lookup table encounters a problem, such as a conflict between a superclass's protocol conformance and a subclass's protocol conformance, it records the problem in a diagnostics side-table and resolves the conflict in a manner that allows other type checking to continue. The actual diagnosis of the problem occurs only when performing complete semantics checking of the declaration that owns the erroneous protocol conformance. - -Note that the conformance lookup table does *not* implement a dependency graph or priority queue as proposed above. Rather, it performs direct recursion internally (which is generally not a problem) and through the current type-validation logic (which requires it to be re-entrant). - -How do we get there? -~~~~~~~~~~~~~~~~~~~~ - -The proposed architecture is significantly different from the current type checker architecture, so how do we get there from here? There are a few concrete steps we can take: - -**Make all AST nodes phase-aware**: Introduce a trait that can ask an arbitrary AST node (``Decl``, ``TypeRepr``, ``Pattern``, etc.) its current phase. AST nodes may compute this information on-the-fly or store it, as appropriate. For example, a ``TypeRepr`` can generally determine its phase based on the existing state of the ``IdentTypeRepr`` nodes it includes. - -**Make name lookup phase-aware**: Name lookup is currently one of the worst offenders when violating phase ordering. Parameterize name lookup based on the phase at which it's operating. For example, asking for name lookup at the "extension binding" phase might not resolve type aliases, look into superclasses, or look into protocols. - -**Make type resolution phase-aware**: Type resolution effectively brings a given ``TypeRepr`` up to the "declaration type validation`` phase in one shot. Parameterize type resolution based on the target phase, and start minimizing the among of work that the type checking does. Use extension binding as a testbed for these more-minimal dependencies. - -**Dependency graph and priority queue**: Extend the current-phase trait with an operation that enumerates the dependencies that need to be satisfied to bring a given AST node up to a particular phase. Start with ``TypeRepr`` nodes, and use the dependency graph and priority queue to satisfy all dependencies ahead of time, eliminating direct recursion from the type-resolution code path. Build circular-dependency detection within this test-bed. - -**Incremental adoption of dependency graph**: Make other AST nodes (``Pattern``, ``VarDecl``, etc.) implement the phase-awareness trait, enumerating dependencies and updating their logic to perform minimal updates. Certain entry points that are used for ad hoc recursion (such as ``validateDecl``) can push/pop dependency graph and priority-queue instances, which leaves the existing ad hoc recursion checking in place but allows isolated subproblems to use the newer mechanisms. - -**Strengthen accessor assertions**: As ad hoc recursion gets eliminated from the type checker, strengthen assertions on the various AST nodes to make sure the AST node has been brought to the appropriate phase. - -How do we test it? -~~~~~~~~~~~~~~~~~~ - -**Existing code continues to work**: As we move various parts of the type checker over to the dependency graph, existing Swift code should continue to work, since we'll have fallbacks to the existing logic and the new type checker should be strictly more lazy than the existing type checker. - -**Order-independence testing**: One of the intended improvements from this type checker architecture is that we should get more predictable order-independent behavior. To check this, we can randomly scramble the order in which we type-check declarations in the primary source file of a well-formed module and verify that we get the same results. - -**Compiler crashers**: The compiler crashers testsuite tends to contain a large number of crashes that are effectively due to infinite recursion in the type checker. We expect that many of these will be resolved. - -How do we measure progress? -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The proposed change is a major architectural shift, and it's only complete when we have eliminated all ad hoc recursion from the front end. There are a few ways in which we can measure progress along the way: - -**AST nodes that implement the phase-aware trait**: Eventually, all of our AST nodes will implement the phase-aware trait. The number of AST nodes that do properly implement that trait (reporting current phase, enumerating dependencies for a phase transition) and become part of the dependency graph and priority queue gives an indication of how far we've gotten. - -**Accessors that check the current phase**: When we're finished, each of the AST's accessors should assert that the AST node is in the appropriate phase. The number of such assertions that have been enabled is an indication of how well the type checker is respecting the dependencies. - -**Phases of AST nodes in non-primary files**: With the current type checker, every AST node in a non-primary file that gets touched when type-checking the primary file will end up being fully validated (currently, the "attribute checking" phase). As the type checker gets more lazy, the AST nodes in non-primary files will trend toward earlier phases. Tracking the number of nodes in non-primary files at each phase over time will help us establish how lazy the type checker is becoming. diff --git a/docs/proposals/EnumStyle.md b/docs/proposals/EnumStyle.md new file mode 100644 index 0000000000000..5c1d813e5374c --- /dev/null +++ b/docs/proposals/EnumStyle.md @@ -0,0 +1,360 @@ +One of the issues that came up in our design discussions around `Result` +was that enum cases don't really follow the conventions of anything else +in our system. Our current convention for enum cases is to use +`CapitalizedCamelCase`. This convention arose from the Cocoa +`NSEnumNameCaseName` convention for constants, but the convention feels +foreign even in the context of Objective-C. Non-enum type constants in +Cocoa are often namespaced into classes, using class methods such as +`[UIColor redColor]` (and would likely have been class properties if +those were supported by ObjC). It's also worth noting that our "builtin" +enum-like keywords such as `true`, `false`, and `nil` are lowercased, +more like properties. + +Swift also has enum cases with associated values, which don't have an +immediate analog in Cocoa to draw inspiration from, but if anything feel +"initializer-like". Aside from naming style, working with enum values +also requires a different set of tools from other types, pattern +matching with `switch` or `if case` instead of working with more +readily-composable expressions. The compound effect of these style +mismatches is that enums in the wild tend to grow a bunch of boilerplate +helper members in order to make them fit better with other types. For +example, `Optional`, aside from the massive amounts of language sugar +it's given, vends initializers corresponding to `Some` and `None` cases: + + extension Optional { + init(_ value: Wrapped) { + self = .Some(value) + } + + init() { + self = .None + } + } + +`Result` was proposed to have not only initializers corresponding to its +`Success` and `Error` cases, but accessor properties as well: + + extension Result { + init(success: Wrapped) { + self = .Success(success) + } + init(error: ErrorType) { + self = .Error(error) + } + + var success: Wrapped? { + switch self { + case .Success(let success): return success + case .Error: return nil + } + } + var error: ErrorType? { + switch self { + case .Success: return nil + case .Error(let error): return error + } + } + } + +This pattern of boilerplate also occurs in third-party frameworks that +make heavy use of enums. Some examples from Github: + +- +- +- + +That people inside and outside of our team consider this boilerplate +necessary for enums is a strong sign we should improve our core language +design. I'd like to start discussion by proposing the following: + +- Because cases with associated values are initializer-like, declaring + and using them ought to feel like using initializers on other types. + A `case` declaration should be able to declare an initializer, which + follows the same keyword naming rules as other initializers, for + example: + + enum Result { + case init(success: Wrapped) + case init(error: ErrorType) + } + + Constructing a value of the case can then be done with the usual + initializer syntax: + + let success = Result(success: 1) + let error = Result(error: SillyError.JazzHands) + + And case initializers can be pattern-matched using initializer-like + matching syntax: + + switch result { + case Result(success: let success): + ... + case Result(error: let error): + ... + } + +- Enums with associated values implicitly receive `internal` + properties corresponding to the argument labels of those + associated values. The properties are optional-typed unless a value + with the same name and type appears in every `case`. For example, + this enum: + + public enum Example { + case init(foo: Int, alwaysPresent: String) + case init(bar: Int, alwaysPresent: String) + } + + receives the following implicit members: + + /*implicit*/ + internal extension Example { + var foo: Int? { get } + var bar: Int? { get } + var alwaysPresent: String { get } // Not optional + } + +- Because cases without associated values are property-like, they + ought to follow the `lowercaseCamelCase` naming convention of + other properties. For example: + + enum ComparisonResult { + case descending, same, ascending + } + + enum Bool { + case true, false + } + + enum Optional { + case nil + case init(_ some: Wrapped) + } + +Since this proposal affects how we name things, it has ABI stability +implications (albeit ones we could hack our way around with enough +symbol aliasing), so I think we should consider this now. It also meshes +with other naming convention discussions that have been happening. + +I'll discuss the points above in more detail: + +Case Initializers +================= + +Our standard recommended style for cases with associated values should +be to declare them as initializers with keyword arguments, much as we do +other kinds of initializer: + + enum Result { + case init(success: Wrapped) + case init(error: ErrorType) + } + + enum List { + case empty + indirect case init(element: Element, rest: List) + } + +It should be possible to declare unlabeled case initializers too, for +types like Optional with a natural "primary" case: + + enum Optional { + case nil + case init(_ some: Wrapped) + } + +Patterns should also be able to match against case initializers: + + switch result { + case Result(success: let s): + ... + case Result(error: let e): + ... + } + +Overloading +----------- + +I think it would also be reasonable to allow overloading of case +initializers, as long as the associated value types cannot overlap. (If +the keyword labels are overloaded and the associated value types +overlap, there would be no way to distinguish the cases.) Overloading is +not essential, though, and it would be simpler to disallow it. + +Named cases with associated values +---------------------------------- + +One question would be, if we allow `case init` declarations, whether we +should also remove the existing ability to declare named cases with +associated values: + + enum Foo { + // OK + case init(foo: Int) + // Should this become an error? + case foo(Int) + } + +Doing so would help unambiguously push the new style, but would drive a +syntactic wedge between associated-value and no-associated-value cases. +If we keep named cases with associated values, I think we should +consider altering the declaration syntax to require keyword labels (or +explicit `_` to suppress labels), for better consistency with other +function-like decls: + + enum Foo { + // Should be a syntax error, 'label:' expected + case foo(Int) + + // OK + case foo(_: Int) + + // OK + case foo(label: Int) + } + +Shorthand for init-style cases +------------------------------ + +Unlike enum cases and static methods, initializers currently don't have +any contextual shorthand when the type of an initialization can be +inferred from context. This could be seen as an expressivity regression +in some cases. With named cases, one can write: + + foo(.Left(x)) + +but with case initializers, they have to write: + + foo(Either(left: x)) + +Some would argue this is clearer. It's a bit more painful in `switch` +patterns, though, where the type would need to be repeated redundantly: + + switch x { + case Either(left: let left): + ... + case Either(right: let right): + ... + } + +One possibility would be to allow `.init`, like we do other static +methods: + + switch x { + case .init(left: let left): + ... + case .init(right: let right): + ... + } + +Or maybe allow labeled tuple patterns to match, leaving the name off +altogether: + + switch x { + case (left: let left): + ... + case (right: let right): + ... + } + +Implicit Case Properties +======================== + +The only native operation enums currently support is `switch`-ing. This +is nice and type-safe, but `switch` is heavyweight and not very +expressive. We now have a large set of language features and library +operators for working with `Optional`, so it is expressive and +convenient in many cases to be able to project associated values from +enums as `Optional` values. As noted above, third-party developers using +enums often write out the boilerplate to do this. We should automate it. +For every `case init` with labeled associated values, we can generate an +`internal` property to access that associated value. The value will be +`Optional`, unless every `case` has the same associated value, in which +case it can be nonoptional. To repeat the above example, this enum: + + public enum Example { + case init(foo: Int, alwaysPresent: String) + case init(bar: Int, alwaysPresent: String) + } + +receives the following implicit members: + + /*implicit*/ + internal extension Example { + var foo: Int? { get } + var bar: Int? { get } + var alwaysPresent: String { get } // Not optional + } + +Similar to the elementwise initializer for `struct` types, these +property accessors should be `internal`, since they rely on potentially +fragile layout characteristics of the enum. (Like the struct elementwise +initializer, we ought to have a way to easily export these properties as +`public` when desired too, but that can be designed separately.) + +These implicit properties should be read-only, until we design a model +for enum mutation-by-part. + +An associated value property should be suppressed if: + +- there's an explicit declaration in the type with the same name: + + enum Foo { + case init(foo: Int) + + var foo: String { return "foo" } // suppresses implicit "foo" property + } + +- there are associated values with the same label but conflicting + types: + + enum Foo { + case init(foo: Int, bar: Int) + case init(foo: String, bas: Int) + + // No 'foo' property, because of conflicting associated values + } + +- if the associated value has no label: + + enum Foo { + case init(_: Int) + + // No property for the associated value + } + + An associated value could be unlabeled but still provide an internal + argument name to name its property: + + enum Foo { + case init(_ x: Int) + case init(_ y: String) + + // var x: Int? + // var y: String? + } + +Naming Conventions for Enum Cases +================================= + +To normalize enums and bring them into the "grand unified theory" of +type interfaces shared by other Swift types, I think we should encourage +the following conventions: + +- Cases with associated values should be declared as `case init` + initializers with labeled associated values. +- Simple cases without associated values should be named like + properties, using `lowercaseCamelCase`. We should also import Cocoa + `NS_ENUM` and `NS_OPTIONS` constants using `lowercaseCamelCase`. + +This is a big change from the status quo, including the Cocoa tradition +for C enum constants, but I think it's the right thing to do. Cocoa uses +the `NSEnumNameCaseName` convention largely because enum constants are +not namespaced in Objective-C. When Cocoa associates constants with +class types, it uses its normal method naming conventions, as in +`UIColor.redColor`. In Swift's standard library, type constants for +structs follow the same convention, for example `Int.max` and `Int.min`. +The literal keywords `true`, `false`, and `nil` are arguably +enum-case-like and also lowercased. Simple enum cases are essentially +static constant properties of their type, so they should follow the same +conventions. diff --git a/docs/proposals/EnumStyle.rst b/docs/proposals/EnumStyle.rst deleted file mode 100644 index e83101e21d30e..0000000000000 --- a/docs/proposals/EnumStyle.rst +++ /dev/null @@ -1,357 +0,0 @@ -:orphan: - -One of the issues that came up in our design discussions around ``Result`` was -that enum cases don't really follow the conventions of anything else in our -system. Our current convention for enum cases is to use -``CapitalizedCamelCase``. This convention arose from the Cocoa -``NSEnumNameCaseName`` convention for constants, but the convention feels -foreign even in the context of Objective-C. Non-enum type constants in Cocoa -are often namespaced into classes, using class methods such as ``[UIColor -redColor]`` (and would likely have been class properties if those were -supported by ObjC). It's also worth noting that our "builtin" enum-like -keywords such as ``true``, ``false``, and ``nil`` are lowercased, more like -properties. - -Swift also has enum cases with associated values, which don't have an immediate -analog in Cocoa to draw inspiration from, but if anything feel -"initializer-like". Aside from naming style, working with enum values also -requires a different set of tools from other types, pattern matching with -``switch`` or ``if case`` instead of working with more readily-composable -expressions. The compound effect of these style mismatches is that enums in the -wild tend to grow a bunch of boilerplate helper members in order to make them -fit better with other types. For example, ``Optional``, aside from the massive -amounts of language sugar it's given, vends initializers corresponding to -``Some`` and ``None`` cases:: - - extension Optional { - init(_ value: Wrapped) { - self = .Some(value) - } - - init() { - self = .None - } - } - -``Result`` was proposed to have not only initializers corresponding to its -``Success`` and ``Error`` cases, but accessor properties as well:: - - extension Result { - init(success: Wrapped) { - self = .Success(success) - } - init(error: ErrorType) { - self = .Error(error) - } - - var success: Wrapped? { - switch self { - case .Success(let success): return success - case .Error: return nil - } - } - var error: ErrorType? { - switch self { - case .Success: return nil - case .Error(let error): return error - } - } - } - -This pattern of boilerplate also occurs in third-party frameworks that make -heavy use of enums. Some examples from Github: - -- https://github.com/antitypical/Manifold/blob/ae94eb96085c2c8195d457e06df485b1cca455cb/Manifold/Name.swift -- https://github.com/antitypical/TesseractCore/blob/73099ae5fa772b90cefa49395f237290d8363f76/TesseractCore/Symbol.swift -- https://github.com/antitypical/TesseractCore/blob/73099ae5fa772b90cefa49395f237290d8363f76/TesseractCore/Value.swift - -That people inside and outside of our team consider this boilerplate necessary -for enums is a strong sign we should improve our core language design. -I'd like to start discussion by proposing the following: - -- Because cases with associated values are initializer-like, declaring and - using them ought to feel like using initializers on other types. - A ``case`` declaration should be able to declare an initializer, which - follows the same keyword naming rules as other initializers, for example:: - - enum Result { - case init(success: Wrapped) - case init(error: ErrorType) - } - - Constructing a value of the case can then be done with the usual initializer - syntax:: - - let success = Result(success: 1) - let error = Result(error: SillyError.JazzHands) - - And case initializers can be pattern-matched using initializer-like - matching syntax:: - - switch result { - case Result(success: let success): - ... - case Result(error: let error): - ... - } - -- Enums with associated values implicitly receive ``internal`` properties - corresponding to the argument labels of those associated values. The - properties are optional-typed unless a value with the same name and type - appears in every ``case``. For example, this enum:: - - public enum Example { - case init(foo: Int, alwaysPresent: String) - case init(bar: Int, alwaysPresent: String) - } - - receives the following implicit members:: - - /*implicit*/ - internal extension Example { - var foo: Int? { get } - var bar: Int? { get } - var alwaysPresent: String { get } // Not optional - } - -- Because cases without associated values are property-like, they ought to - follow the ``lowercaseCamelCase`` naming convention of other properties. - For example:: - - enum ComparisonResult { - case descending, same, ascending - } - - enum Bool { - case true, false - } - - enum Optional { - case nil - case init(_ some: Wrapped) - } - -Since this proposal affects how we name things, it has ABI stability -implications (albeit ones we could hack our way around with enough symbol -aliasing), so I think we should consider this now. It also meshes with other -naming convention discussions that have been happening. - -I'll discuss the points above in more detail: - -Case Initializers -================= - -Our standard recommended style for cases with associated values should be -to declare them as initializers with keyword arguments, much as we do -other kinds of initializer:: - - enum Result { - case init(success: Wrapped) - case init(error: ErrorType) - } - - enum List { - case empty - indirect case init(element: Element, rest: List) - } - -It should be possible to declare unlabeled case initializers too, for types -like Optional with a natural "primary" case:: - - enum Optional { - case nil - case init(_ some: Wrapped) - } - -Patterns should also be able to match against case initializers:: - - switch result { - case Result(success: let s): - ... - case Result(error: let e): - ... - } - -Overloading ------------ - -I think it would also be reasonable to allow overloading of case initializers, -as long as the associated value types cannot overlap. (If the keyword labels -are overloaded and the associated value types overlap, there would -be no way to distinguish the cases.) Overloading is not essential, though, and -it would be simpler to disallow it. - -Named cases with associated values ----------------------------------- - -One question would be, if we allow ``case init`` declarations, whether we -should also remove the existing ability to declare named cases with associated -values:: - - enum Foo { - // OK - case init(foo: Int) - // Should this become an error? - case foo(Int) - } - -Doing so would help unambiguously push the new style, but would drive a -syntactic wedge between associated-value and no-associated-value cases. -If we keep named cases with associated values, I think we should consider -altering the declaration syntax to require keyword labels (or explicit ``_`` -to suppress labels), for better consistency with other function-like decls:: - - enum Foo { - // Should be a syntax error, 'label:' expected - case foo(Int) - - // OK - case foo(_: Int) - - // OK - case foo(label: Int) - } - -Shorthand for init-style cases ------------------------------- - -Unlike enum cases and static methods, initializers currently don't have any -contextual shorthand when the type of an initialization can be inferred from -context. This could be seen as an expressivity regression in some cases. -With named cases, one can write:: - - foo(.Left(x)) - -but with case initializers, they have to write:: - - foo(Either(left: x)) - -Some would argue this is clearer. It's a bit more painful in ``switch`` -patterns, though, where the type would need to be repeated redundantly:: - - switch x { - case Either(left: let left): - ... - case Either(right: let right): - ... - } - -One possibility would be to allow ``.init``, like we do other static methods:: - - switch x { - case .init(left: let left): - ... - case .init(right: let right): - ... - } - -Or maybe allow labeled tuple patterns to match, leaving the name off -altogether:: - - switch x { - case (left: let left): - ... - case (right: let right): - ... - } - -Implicit Case Properties -======================== - -The only native operation enums currently support is ``switch``-ing. This is -nice and type-safe, but ``switch`` is heavyweight and not very expressive. -We now have a large set of language features and library operators for working -with ``Optional``, so it is expressive and convenient in many cases to be able -to project associated values from enums as ``Optional`` values. As noted above, -third-party developers using enums often write out the boilerplate to do this. -We should automate it. For every ``case init`` with labeled associated values, -we can generate an ``internal`` property to access that associated value. -The value will be ``Optional``, unless every ``case`` has the same associated -value, in which case it can be nonoptional. To repeat the above example, this -enum:: - - public enum Example { - case init(foo: Int, alwaysPresent: String) - case init(bar: Int, alwaysPresent: String) - } - -receives the following implicit members:: - - /*implicit*/ - internal extension Example { - var foo: Int? { get } - var bar: Int? { get } - var alwaysPresent: String { get } // Not optional - } - -Similar to the elementwise initializer for ``struct`` types, these property -accessors should be ``internal``, since they rely on potentially fragile layout -characteristics of the enum. (Like the struct elementwise initializer, we -ought to have a way to easily export these properties as ``public`` when -desired too, but that can be designed separately.) - -These implicit properties should be read-only, until we design a model for -enum mutation-by-part. - -An associated value property should be suppressed if: - -- there's an explicit declaration in the type with the same name:: - - enum Foo { - case init(foo: Int) - - var foo: String { return "foo" } // suppresses implicit "foo" property - } - -- there are associated values with the same label but conflicting types:: - - enum Foo { - case init(foo: Int, bar: Int) - case init(foo: String, bas: Int) - - // No 'foo' property, because of conflicting associated values - } - -- if the associated value has no label:: - - enum Foo { - case init(_: Int) - - // No property for the associated value - } - - An associated value could be unlabeled but still provide an internal argument - name to name its property:: - - enum Foo { - case init(_ x: Int) - case init(_ y: String) - - // var x: Int? - // var y: String? - } - -Naming Conventions for Enum Cases -================================= - -To normalize enums and bring them into the "grand unified theory" of type -interfaces shared by other Swift types, I think we should encourage the -following conventions: - -- Cases with associated values should be declared as ``case init`` - initializers with labeled associated values. -- Simple cases without associated values should be named like properties, - using ``lowercaseCamelCase``. We should also import Cocoa ``NS_ENUM`` - and ``NS_OPTIONS`` constants using ``lowercaseCamelCase``. - -This is a big change from the status quo, including the Cocoa tradition for -C enum constants, but I think it's the right thing to do. Cocoa uses -the ``NSEnumNameCaseName`` convention largely because enum constants are -not namespaced in Objective-C. When Cocoa associates constants with -class types, it uses its normal method naming conventions, as in -``UIColor.redColor``. In Swift's standard library, type constants for structs -follow the same convention, for example ``Int.max`` and ``Int.min``. The -literal keywords ``true``, ``false``, and ``nil`` are arguably enum-case-like -and also lowercased. Simple enum cases are essentially static constant -properties of their type, so they should follow the same conventions. - diff --git a/docs/proposals/Enums.md b/docs/proposals/Enums.md new file mode 100644 index 0000000000000..1b7a6a56279a3 --- /dev/null +++ b/docs/proposals/Enums.md @@ -0,0 +1,349 @@ +Swift supports what type theory calls "algebraic data types", or ADTs, +which are an amalgam of two familiar C-family language features, enums +and unions. They are similar to enums in that they allow collections of +independent symbolic values to be collected into a type and switched +over: + + enum Color { + case Red, Green, Blue, Black, White + } + + var c : Color = .Red + switch c { + case .Red: + ... + case .Green: + ... + case .Blue: + ... + } + +They are also similar to C unions in that they allow a single type to +contain a value of two or more other types. Unlike C unions, however, +ADTs remember which type they contain, and can be switched over, +guaranteeing that only the currently inhabited type is ever used: + + enum Pattern { + case Solid(Color) + case Outline(Color) + case Checkers(Color, Color) + } + + var p : Pattern = .Checkers(.Black, .White) + switch p { + case .Solid(var c): + print("solid \(c)") + case .Outline(var c): + print("outlined \(c)") + case .Checkers(var a, var b): + print("checkered \(a) and \(b)") + } + +Given the choice between two familiar keywords, we decided to use 'enum' +to name these types. Here are some of the reasons why: + +Why 'enum'? +=========== + +The common case works like C +---------------------------- + +C programmers with no interest in learning about ADTs can use 'enum' +like they always have. + +"Union" doesn't exist to Cocoa programmers +------------------------------------------ + +Cocoa programmers really don't think about unions at all. The frameworks +vend no public unions. If a Cocoa programmer knows what a union is, it's +as a broken C bit-bangy thing. Cocoa programmers are used to more safely +and idiomatically modeling ADTs in Objective-C as class hierarchies. The +concept of closed-hierarchy variant value types is new territory for +them, so we have some freedom in choosing how to present the feature. +Trying to relate it to C's 'union', a feature with negative +connotations, is a disservice if anything that will dissuade users from +wanting to learn and take advantage of it. + +It parallels our extension of 'switch' +-------------------------------------- + +The idiomatic relationship between 'enum' and 'switch' in C is +well-established--If you have an enum, the best practice for consuming +it is to switch over it so the compiler can check exhaustiveness for +you. We've extended 'switch' with pattern matching, another new concept +for our target audience, and one that happens to be dual to the concept +of enums with payload. In the whitepaper, we introduce pattern matching +by starting from the familiar C case of switching over an integer and +gradually introduce the new capabilities of Swift's switch. If all ADTs +are 'enums', this lets us introduce both features to C programmers +organically, starting from the familiar case that looks like C: + + enum Foo { case A, B, C, D } + + func use(x:Foo) { + switch x { + case .A: + case .B: + case .C: + case .D: + } + } + +and then introducing the parallel new concepts of payloads and patterns +together: + + enum Foo { case A, B, C, D, Other(String) } + + func use(x:Foo) { + switch x { + case .A: + case .B: + case .C: + case .D: + case .Other(var s): + } + } + +People already use 'enum' to define ADTs, badly +----------------------------------------------- + +Enums are already used and abused in C in various ways as a building +block for ADT-like types. An enum is of course the obvious choice to +represented the discriminator in a tagged-union structure. Instead of +saying 'you write union and get the enum for free', we can switch the +message around: 'you write enum and get the union for free'. Aside from +that case, though, there are many uses in C of enums as ordered +integer-convertible values that are really trying to express more +complex symbolic ADTs. For example, there's the pervasive LLVM +convention of 'First\_*' and 'Last\_*' sigils: + + /* C */ + enum Pet { + First_Reptile, + Lizard = First_Reptile, + Snake, + Last_Reptile = Snake, + + First_Mammal, + Cat = First_Mammal, + Dog, + Last_Mammal = Dog, + }; + +which is really crying out for a nested ADT representation: + + // Swift + enum Reptile { case Lizard, Snake } + enum Mammal { case Cat, Dog } + enum Pet { + case Reptile(Reptile) + case Mammal(Mammal) + } + +Or there's the common case of an identifier with standardized symbolic +values and a 'user-defined' range: + + /* C */ + enum Language : uint16_t { + C89, + C99, + Cplusplus98, + Cplusplus11, + First_UserDefined = 0x8000, + Last_UserDefined = 0xFFFF + }; + +which again is better represented as an ADT: + + // Swift + enum Language { + case C89, C99, Cplusplus98, Cplusplus11 + case UserDefined(UInt16) + } + +Rust does it +------------ + +Rust also labels their ADTs 'enum', so there is some alignment with the +"extended family" of C-influenced modern systems programming languages +in making the same choice + +Design +====== + +Syntax +------ + +The 'enum' keyword introduces an ADT (hereon called an "enum"). Within +an enum, the 'case' keyword introduces a value of the enum. This can +either be a purely symbolic case or can declare a payload type that is +stored with the value: + + enum Color { + case Red + case Green + case Blue + } + + enum Optional { + case Some(T) + case None + } + + enum IntOrInfinity { + case Int(Int) + case NegInfinity + case PosInfinity + } + +Multiple 'case' declarations may be specified in a single declaration, +separated by commas: + + enum IntOrInfinity { + case NegInfinity, Int(Int), PosInfinity + } + +Enum declarations may also contain the same sorts of nested declarations +as structs, including nested types, methods, constructors, and +properties: + + enum IntOrInfinity { + case NegInfinity, Int(Int), PosInfinity + + constructor() { + this = .Int(0) + } + + func min(x:IntOrInfinity) -> IntOrInfinity { + switch (self, x) { + case (.NegInfinity, _): + case (_, .NegInfinity): + return .NegInfinity + case (.Int(var a), .Int(var b)): + return min(a, b) + case (.Int(var a), .PosInfinity): + return a + case (.PosInfinity, .Int(var b)): + return b + } + } + } + +They may not however contain physical properties. + +Enums do not have default constructors (unless one is explicitly +declared). Enum values are constructed by referencing one of its cases, +which are scoped as if static values inside the enum type: + + var red = Color.Red + var zero = IntOrInfinity.Int(0) + var inf = IntOrInfinity.PosInfinity + +If the enum type can be deduced from context, it can be elided and the +case can be referenced using leading dot syntax: + + var inf : IntOrInfinity = .PosInfinity + return inf.min(.NegInfinity) + +The 'RawRepresentable' protocol +------------------------------- + +In the library, we define a compiler-blessed 'RawRepresentable' protocol +that models the traditional relationship between a C enum and its raw +type: + + protocol RawRepresentable { + /// The raw representation type. + typealias RawType + + /// Convert the conforming type to its raw type. + /// Every valid value of the conforming type should map to a unique + /// raw value. + func toRaw() -> RawType + + /// Convert a value of raw type to the corresponding value of the + /// conforming type. + /// Returns None if the raw value has no corresponding conforming type + /// value. + class func fromRaw(_:RawType) -> Self? + } + +Any type may manually conform to the RawRepresentable protocol following +the above invariants, regardless of whether it supports compiler +derivation as underlined below. + +Deriving the 'RawRepresentable' protocol for enums +-------------------------------------------------- + +An enum can obtain a compiler-derived 'RawRepresentable' conformance by +declaring "inheritance" from its raw type in the following +circumstances: + +- The inherited raw type must be IntegerLiteralConvertible, + FloatLiteralConvertible, CharLiteralConvertible, + and/or StringLiteralConvertible. +- None of the cases of the enum may have non-void payloads. + +If an enum declares an raw type, then its cases may declare raw values. +raw values must be integer, float, character, or string literals, and +must be unique within the enum. If the raw type is +IntegerLiteralConvertible, then the raw values default to +auto-incrementing integer literal values starting from '0', as in C. If +the raw type is not IntegerLiteralConvertible, the raw values must all +be explicitly declared: + + enum Color : Int { + case Black // = 0 + case Cyan // = 1 + case Magenta // = 2 + case White // = 3 + } + + enum Signal : Int32 { + case SIGKILL = 9, SIGSEGV = 11 + } + + enum NSChangeDictionaryKey : String { + // All raw values are required because String is not + // IntegerLiteralConvertible + case NSKeyValueChangeKindKey = "NSKeyValueChangeKindKey" + case NSKeyValueChangeNewKey = "NSKeyValueChangeNewKey" + case NSKeyValueChangeOldKey = "NSKeyValueChangeOldKey" + } + +The compiler, on seeing a valid raw type for an enum, derives a +RawRepresentable conformance, using 'switch' to implement the fromRaw +and toRaw methods. The NSChangeDictionaryKey definition behaves as if +defined: + + enum NSChangeDictionaryKey : RawRepresentable { + typealias RawType = String + + case NSKeyValueChangeKindKey + case NSKeyValueChangeNewKey + case NSKeyValueChangeOldKey + + func toRaw() -> String { + switch self { + case .NSKeyValueChangeKindKey: + return "NSKeyValueChangeKindKey" + case .NSKeyValueChangeNewKey: + return "NSKeyValueChangeNewKey" + case .NSKeyValueChangeOldKey: + return "NSKeyValueChangeOldKey" + } + } + + static func fromRaw(s:String) -> NSChangeDictionaryKey? { + switch s { + case "NSKeyValueChangeKindKey": + return .NSKeyValueChangeKindKey + case "NSKeyValueChangeNewKey": + return .NSKeyValueChangeNewKey + case "NSKeyValueChangeOldKey": + return .NSKeyValueChangeOldKey + default: + return nil + } + } + } diff --git a/docs/proposals/Enums.rst b/docs/proposals/Enums.rst deleted file mode 100644 index 46402160b1a1f..0000000000000 --- a/docs/proposals/Enums.rst +++ /dev/null @@ -1,346 +0,0 @@ -:orphan: - -Swift supports what type theory calls "algebraic data types", or ADTs, which -are an amalgam of two familiar C-family language features, enums and unions. -They are similar to enums in that they allow collections of independent symbolic -values to be collected into a type and switched over:: - - enum Color { - case Red, Green, Blue, Black, White - } - - var c : Color = .Red - switch c { - case .Red: - ... - case .Green: - ... - case .Blue: - ... - } - -They are also similar to C unions in that they allow a single type to -contain a value of two or more other types. Unlike C unions, however, ADTs -remember which type they contain, and can be switched over, guaranteeing that -only the currently inhabited type is ever used:: - - enum Pattern { - case Solid(Color) - case Outline(Color) - case Checkers(Color, Color) - } - - var p : Pattern = .Checkers(.Black, .White) - switch p { - case .Solid(var c): - print("solid \(c)") - case .Outline(var c): - print("outlined \(c)") - case .Checkers(var a, var b): - print("checkered \(a) and \(b)") - } - -Given the choice between two familiar keywords, we decided to use 'enum' to -name these types. Here are some of the reasons why: - -Why 'enum'? -=========== - -The common case works like C ----------------------------- - -C programmers with no interest in learning about ADTs can use 'enum' like they -always have. - -"Union" doesn't exist to Cocoa programmers ------------------------------------------- - -Cocoa programmers really don't think about unions at all. The frameworks vend -no public unions. If a Cocoa programmer knows what a union is, it's as a -broken C bit-bangy thing. Cocoa programmers are used to more safely -and idiomatically modeling ADTs in Objective-C as class hierarchies. The -concept of closed-hierarchy variant value types is new territory for them, so -we have some freedom in choosing how to present the feature. Trying to relate -it to C's 'union', a feature with negative connotations, is a disservice if -anything that will dissuade users from wanting to learn and take advantage of -it. - -It parallels our extension of 'switch' --------------------------------------- - -The idiomatic relationship between 'enum' and 'switch' in C is -well-established--If you have an enum, the best practice for consuming it is to -switch over it so the compiler can check exhaustiveness for you. We've extended -'switch' with pattern matching, another new concept for our target audience, -and one that happens to be dual to the concept of enums with payload. In the -whitepaper, we introduce pattern matching by starting from the familiar C case -of switching over an integer and gradually introduce the new capabilities of -Swift's switch. If all ADTs are 'enums', this lets us introduce both features -to C programmers organically, starting from the familiar case that looks like -C:: - - enum Foo { case A, B, C, D } - - func use(x:Foo) { - switch x { - case .A: - case .B: - case .C: - case .D: - } - } - -and then introducing the parallel new concepts of payloads and patterns -together:: - - enum Foo { case A, B, C, D, Other(String) } - - func use(x:Foo) { - switch x { - case .A: - case .B: - case .C: - case .D: - case .Other(var s): - } - } - -People already use 'enum' to define ADTs, badly ------------------------------------------------ - -Enums are already used and abused in C in various ways as a building block for -ADT-like types. An enum is of course the obvious choice to represented the -discriminator in a tagged-union structure. Instead of saying 'you write union -and get the enum for free', we can switch the message around: 'you write enum -and get the union for free'. Aside from that case, though, there are many uses -in C of enums as ordered integer-convertible values that are really trying to -express more complex symbolic ADTs. For example, there's the pervasive LLVM -convention of 'First_*' and 'Last_*' sigils:: - - /* C */ - enum Pet { - First_Reptile, - Lizard = First_Reptile, - Snake, - Last_Reptile = Snake, - - First_Mammal, - Cat = First_Mammal, - Dog, - Last_Mammal = Dog, - }; - -which is really crying out for a nested ADT representation:: - - // Swift - enum Reptile { case Lizard, Snake } - enum Mammal { case Cat, Dog } - enum Pet { - case Reptile(Reptile) - case Mammal(Mammal) - } - -Or there's the common case of an identifier with standardized symbolic values -and a 'user-defined' range:: - - /* C */ - enum Language : uint16_t { - C89, - C99, - Cplusplus98, - Cplusplus11, - First_UserDefined = 0x8000, - Last_UserDefined = 0xFFFF - }; - -which again is better represented as an ADT:: - - // Swift - enum Language { - case C89, C99, Cplusplus98, Cplusplus11 - case UserDefined(UInt16) - } - -Rust does it ------------- - -Rust also labels their ADTs 'enum', so there is some alignment with the -"extended family" of C-influenced modern systems programming languages in making -the same choice - -Design -====== - -Syntax ------- - -The 'enum' keyword introduces an ADT (hereon called an "enum"). Within an enum, -the 'case' keyword introduces a value of the enum. This can either be a purely -symbolic case or can declare a payload type that is stored with the value:: - - enum Color { - case Red - case Green - case Blue - } - - enum Optional { - case Some(T) - case None - } - - enum IntOrInfinity { - case Int(Int) - case NegInfinity - case PosInfinity - } - -Multiple 'case' declarations may be specified in a single declaration, separated -by commas:: - - enum IntOrInfinity { - case NegInfinity, Int(Int), PosInfinity - } - -Enum declarations may also contain the same sorts of nested declarations as -structs, including nested types, methods, constructors, and properties:: - - enum IntOrInfinity { - case NegInfinity, Int(Int), PosInfinity - - constructor() { - this = .Int(0) - } - - func min(x:IntOrInfinity) -> IntOrInfinity { - switch (self, x) { - case (.NegInfinity, _): - case (_, .NegInfinity): - return .NegInfinity - case (.Int(var a), .Int(var b)): - return min(a, b) - case (.Int(var a), .PosInfinity): - return a - case (.PosInfinity, .Int(var b)): - return b - } - } - } - -They may not however contain physical properties. - -Enums do not have default constructors (unless one is explicitly declared). -Enum values are constructed by referencing one of its cases, which are scoped -as if static values inside the enum type:: - - var red = Color.Red - var zero = IntOrInfinity.Int(0) - var inf = IntOrInfinity.PosInfinity - -If the enum type can be deduced from context, it can be elided and the case -can be referenced using leading dot syntax:: - - var inf : IntOrInfinity = .PosInfinity - return inf.min(.NegInfinity) - -The 'RawRepresentable' protocol -------------------------------- - -In the library, we define a compiler-blessed 'RawRepresentable' protocol that -models the traditional relationship between a C enum and its raw type:: - - protocol RawRepresentable { - /// The raw representation type. - typealias RawType - - /// Convert the conforming type to its raw type. - /// Every valid value of the conforming type should map to a unique - /// raw value. - func toRaw() -> RawType - - /// Convert a value of raw type to the corresponding value of the - /// conforming type. - /// Returns None if the raw value has no corresponding conforming type - /// value. - class func fromRaw(_:RawType) -> Self? - } - -Any type may manually conform to the RawRepresentable protocol following the above -invariants, regardless of whether it supports compiler derivation as underlined -below. - -Deriving the 'RawRepresentable' protocol for enums --------------------------------------------------- - -An enum can obtain a compiler-derived 'RawRepresentable' conformance by -declaring "inheritance" from its raw type in the following -circumstances: - -- The inherited raw type must be IntegerLiteralConvertible, - FloatLiteralConvertible, CharLiteralConvertible, and/or - StringLiteralConvertible. -- None of the cases of the enum may have non-void payloads. - -If an enum declares an raw type, then its cases may declare raw -values. raw values must be integer, float, character, or string -literals, and must be unique within the enum. If the raw type is -IntegerLiteralConvertible, then the raw values default to -auto-incrementing integer literal values starting from '0', as in C. If the -raw type is not IntegerLiteralConvertible, the raw values must -all be explicitly declared:: - - enum Color : Int { - case Black // = 0 - case Cyan // = 1 - case Magenta // = 2 - case White // = 3 - } - - enum Signal : Int32 { - case SIGKILL = 9, SIGSEGV = 11 - } - - enum NSChangeDictionaryKey : String { - // All raw values are required because String is not - // IntegerLiteralConvertible - case NSKeyValueChangeKindKey = "NSKeyValueChangeKindKey" - case NSKeyValueChangeNewKey = "NSKeyValueChangeNewKey" - case NSKeyValueChangeOldKey = "NSKeyValueChangeOldKey" - } - -The compiler, on seeing a valid raw type for an enum, derives a RawRepresentable -conformance, using 'switch' to implement the fromRaw and toRaw -methods. The NSChangeDictionaryKey definition behaves as if defined:: - - enum NSChangeDictionaryKey : RawRepresentable { - typealias RawType = String - - case NSKeyValueChangeKindKey - case NSKeyValueChangeNewKey - case NSKeyValueChangeOldKey - - func toRaw() -> String { - switch self { - case .NSKeyValueChangeKindKey: - return "NSKeyValueChangeKindKey" - case .NSKeyValueChangeNewKey: - return "NSKeyValueChangeNewKey" - case .NSKeyValueChangeOldKey: - return "NSKeyValueChangeOldKey" - } - } - - static func fromRaw(s:String) -> NSChangeDictionaryKey? { - switch s { - case "NSKeyValueChangeKindKey": - return .NSKeyValueChangeKindKey - case "NSKeyValueChangeNewKey": - return .NSKeyValueChangeNewKey - case "NSKeyValueChangeOldKey": - return .NSKeyValueChangeOldKey - default: - return nil - } - } - } - diff --git a/docs/proposals/Error Handling.md b/docs/proposals/Error Handling.md new file mode 100644 index 0000000000000..f36272ef162f6 --- /dev/null +++ b/docs/proposals/Error Handling.md @@ -0,0 +1,153 @@ +Swift Error Handling Model +========================== + +The goal of this writeup is to capture ideas around error handling in +APIs and the tradeoffs involved that influenced this design. + +Error Handling In Contemporary Languages +---------------------------------------- + +C and POSIX proffer a mix of "errno" based APIs and APIs that return an +error code explicitly as a result. While this approach \_works\_, it has +a number of problems: 1) it is too easy to accidentally ignore an error, +2) using a thread local integer variable for error handling is +non-extensible, and 3) using the return value as an error code in C +forces the logical result of the function to be returned by reference +through an argument pointer. + +In contrast, the "obvious" approach to error handling is exception +handling as known from C++, Java, C\#, and many other languages. +Exception handling allows decoupling the logic that produces an error, +the (implicitly generated) logic that propagates the error, and the +logic that ultimately handles the error code. The implementation model +allows a choice of either "zero cost" exceptions which have a slow error +case, or an implicitly generated propagation which slows down the normal +case a bit to make error propagation faster. That said, there are a lot +of ways to do exception handling wrong. + +### Exception Specifications + +Exception specifications are difficult to get right. Adding an concrete +exception specification (i.e., "I only throw T") to a function is a very +strong guarantee that is often difficult to maintain as APIs evolve. For +this reason, Java has two sorts of exceptions: normal ones that obey +exception specifications and "runtime" exceptions that are exempt from +them. In practice, this loop-hole makes exception specifications in Java +completely useless for reasoning about the exception behavior of a +program, which is one of the reasons that C\# eliminated them +completely. + +C++'98 has other unfortunate issues with its exception specifications, +including that all functions are assumed to implicitly throw if they +have no exception spec. Also, its design for empty exception +specification (e.g. "void foo() throw()") is also problematic, and was +improved by adding "noexcept" in C++'11. + +Objective-C is interesting because its exception specifications (i.e. +the presence of an NSError\*\* argument) is binary: a function can +return an error or not, but it isn't encouraged to document *how* APIs +can fail in detailed ways. + +### Runtime Failures + +Both Objective-C and Java recognize a difference between general +application errors and "runtime" errors (such as out-of-bound NSArray +accesses, or a null pointer dereference in Java). As mentioned above, +Java allows runtime errors to avoid exception specifications, but +otherwise treats them the same as other exceptions. + +Objective-C handles runtime errors by throwing an Objective-C exception, +which is a somewhat hard failure because very little Objective-C code is +exception safe. This leads to memory leaks or have other adverse +effects, which is not regarded as recoverable behavior. + +A unfortunate aspect of allowing runtime exceptions to be "caught" is +that it means that removing "guard rails" in the language (e.g. turning +off array bounds checks or null pointer dereference checks) can turn a +working application (one that detects, catches, and handles the error) +into a broken application (one that scribbles on garbage memory). + +### Other Problems with Exception Handling + +C++'s exception handling model causes many systems applications (e.g., +LLVM, Webkit, and many others) to disable exception handling with +-fno-exception. One issue is that C++ exception handling violates its +own "pay for what you use" model of C++ by bloating your code with +exception tables and RTTI data, even if you don't actually throw any +exceptions. The fact that C++ has a poor model to reason about what +calls actually throw also leads to pessimization in the optimizer as it +has to assume the worst case about exception edges, leading to lower +performance (even with "zero cost" exceptions) and code bloat compared +to building with -fno-exceptions. + +C++ also requires a very specific design style (emphasizing RAII) to +make an application exception safe because it lacks automatic memory +management. + +Another common reason that C++ code disables exceptions is that they +want a more "disciplined" or "strict" mode for writing their code. Many +people (particularly at the lower levels of "systems programming" stack) +want to know about and reason about the error handling and propagation +behavior of every error state that can happen, and do not want the +implicit propagation aspect of exceptions. + +Finally, because a lot of disables exceptions, many libraries actively +avoid designing them into their APIs. The STL in particular has very few +APIs that throw exceptions on error cases, and those APIs have +non-throwing counterparts. + +Error Handling Goals +-------------------- + +The design of an error handling system has conflicting goals based on +the audience: some programmers don't want to think about error handling +logic at all - yielding a more "scripting language" sort of experience, +while some people want to control every error case and be forced to +think about error handling in depth - yielding a more "disciplined" +experience. Neither of these is "wrong" or better than the other, they +serve different needs and Swift should support both use cases. + +While level of strictness is negotiable and Swift should support +multiple approaches, the error handling behavior of stable *API* is +something that must be considered as strongly as the arguments and +return value of the function. We consider it a breaking change (and +therefore, unacceptable) for API that was previously guaranteed to never +return an error to start returning error codes. + +It's worth noting that Objective-C achieves these goals with NSError. +NSError "results" are explicitly part of the signature of a method, and +one cannot be added or removed without changing the selector (a breaking +change). Clients who don't care about error handling can (and often do) +completely ignore the NSError result of a method call. + +Swift Error Handling Model +-------------------------- + +Swift categorizes error conditions into two classifications: exceptions +and unrecoverable runtime errors. Either condition can be raised by +arbitrary code, but the two are implemented in different ways and have +different ramifications for propagation and handling of the condition. + +### Swift Runtime Errors + +Runtime errors are conditions like deferencing a null pointer, accessing +an array out of bounds, and explicitly declared exceptions (e.g. out of +memory conditions, at least in some cases). Because they can occur +anywhere, they are not explicitly declared as part of API - any function +is assumed to be capable of raising a runtime error. + +are considered to be "uncatchable" failures that terminate the current +thread/actor, and are + +Runtime errors can occur anywhere in the application + +array out of bounds assertion failure, pre/post conditions failures, +typestate violation. cast<T>(V) + +### Swift Exceptions + +TODO + +strict mode, vs sloppy mode. + +API means something is strict. diff --git a/docs/proposals/Error Handling.rst b/docs/proposals/Error Handling.rst deleted file mode 100644 index 697b9d3b406aa..0000000000000 --- a/docs/proposals/Error Handling.rst +++ /dev/null @@ -1,171 +0,0 @@ -:orphan: - -.. @raise litre.TestsAreMissing -.. ErrorHandlingModel: - -Swift Error Handling Model -========================== - -The goal of this writeup is to capture ideas around error handling in APIs and -the tradeoffs involved that influenced this design. - -Error Handling In Contemporary Languages ----------------------------------------- - -C and POSIX proffer a mix of "errno" based APIs and APIs that return an error -code explicitly as a result. While this approach _works_, it has a number of -problems: 1) it is too easy to accidentally ignore an error, 2) using a thread -local integer variable for error handling is non-extensible, and 3) using the -return value as an error code in C forces the logical result of the function to -be returned by reference through an argument pointer. - -In contrast, the "obvious" approach to error handling is exception handling as -known from C++, Java, C#, and many other languages. Exception handling allows -decoupling the logic that produces an error, the (implicitly generated) -logic that propagates the error, and the logic that ultimately handles the error -code. The implementation model allows a choice of either "zero cost" exceptions -which have a slow error case, or an implicitly generated propagation which slows -down the normal case a bit to make error propagation faster. That said, there -are a lot of ways to do exception handling wrong. - -Exception Specifications -```````````````````````` - -Exception specifications are difficult to get right. Adding an concrete -exception specification (i.e., "I only throw T") to a function is a very strong -guarantee that is often difficult to maintain as APIs evolve. For this reason, -Java has two sorts of exceptions: normal ones that obey exception specifications -and "runtime" exceptions that are exempt from them. In practice, this loop-hole -makes exception specifications in Java completely useless for reasoning about -the exception behavior of a program, which is one of the reasons that C# -eliminated them completely. - -C++'98 has other unfortunate issues with its exception specifications, including -that all functions are assumed to implicitly throw if they have no exception -spec. Also, its design for empty exception specification (e.g. -"void foo() throw()") is also problematic, and was improved by adding "noexcept" -in C++'11. - -Objective-C is interesting because its exception specifications (i.e. the -presence of an NSError** argument) is binary: a function can return an -error or not, but it isn't encouraged to document *how* APIs can fail in -detailed ways. - -Runtime Failures -```````````````` - -Both Objective-C and Java recognize a difference between general application -errors and "runtime" errors (such as out-of-bound NSArray accesses, or a null -pointer dereference in Java). As mentioned above, Java allows runtime errors -to avoid exception specifications, but otherwise treats them the same as other -exceptions. - -Objective-C handles runtime errors by throwing an Objective-C exception, which -is a somewhat hard failure because very little Objective-C code is exception -safe. This leads to memory leaks or have other adverse effects, which is not -regarded as recoverable behavior. - -A unfortunate aspect of allowing runtime exceptions to be "caught" is that it -means that removing "guard rails" in the language (e.g. turning off array bounds -checks or null pointer dereference checks) can turn a working application (one -that detects, catches, and handles the error) into a broken application (one -that scribbles on garbage memory). - - -Other Problems with Exception Handling -`````````````````````````````````````` - -C++'s exception handling model causes many systems applications (e.g., LLVM, -Webkit, and many others) to disable exception handling with -fno-exception. One -issue is that C++ exception handling violates its own "pay for what you use" -model of C++ by bloating your code with exception tables and RTTI data, -even if you don't actually throw any exceptions. The fact that C++ has a poor -model to reason about what calls actually throw also leads to pessimization in -the optimizer as it has to assume the worst case about exception edges, leading -to lower performance (even with "zero cost" exceptions) and code bloat compared -to building with -fno-exceptions. - -C++ also requires a very specific design style (emphasizing RAII) to make an -application exception safe because it lacks automatic memory management. - -Another common reason that C++ code disables exceptions is that they want a more -"disciplined" or "strict" mode for writing their code. Many people -(particularly at the lower levels of "systems programming" stack) want to know -about and reason about the error handling and propagation behavior of every -error state that can happen, and do not want the implicit propagation aspect of -exceptions. - -Finally, because a lot of disables exceptions, many libraries actively avoid -designing them into their APIs. The STL in particular has very few APIs that -throw exceptions on error cases, and those APIs have non-throwing counterparts. - -Error Handling Goals --------------------- - -The design of an error handling system has conflicting goals based on the -audience: some programmers don't want to think about error handling logic at -all - yielding a more "scripting language" sort of experience, while some people -want to control every error case and be forced to think about error handling in -depth - yielding a more "disciplined" experience. Neither of these is "wrong" -or better than the other, they serve different needs and Swift should support -both use cases. - -While level of strictness is negotiable and Swift should support multiple -approaches, the error handling behavior of stable *API* is something that must -be considered as strongly as the arguments and return value of the function. We -consider it a breaking change (and therefore, unacceptable) for API that was -previously guaranteed to never return an error to start returning error codes. - -It's worth noting that Objective-C achieves these goals -with NSError. NSError "results" are explicitly part of the signature of a -method, and one cannot be added or removed without changing the selector (a -breaking change). Clients who don't care about error handling can (and often -do) completely ignore the NSError result of a method call. - - -Swift Error Handling Model --------------------------- - -Swift categorizes error conditions into two classifications: exceptions and -unrecoverable runtime errors. Either condition can be raised by arbitrary code, -but the two are implemented in different ways and have different ramifications -for propagation and handling of the condition. - -Swift Runtime Errors -```````````````````` - -Runtime errors are conditions like deferencing a null pointer, accessing an -array out of bounds, and explicitly declared exceptions (e.g. out of memory -conditions, at least in some cases). Because they can occur anywhere, they are -not explicitly declared as part of API - any function is assumed to be capable -of raising a runtime error. - -are considered to be "uncatchable" failures that terminate the -current thread/actor, and are - - -Runtime errors can occur anywhere in the application - -array out of bounds -assertion failure, pre/post conditions failures, typestate violation. -cast(V) - - - - - - - - -Swift Exceptions -```````````````` - - -TODO - - -strict mode, vs sloppy mode. - -API means something is strict. - - diff --git a/docs/proposals/Initialization.md b/docs/proposals/Initialization.md new file mode 100644 index 0000000000000..6cdb03aa25efe --- /dev/null +++ b/docs/proposals/Initialization.md @@ -0,0 +1,334 @@ +Initialization +============== + +Superclass Delegation +--------------------- + +The initializer for a class that has a superclass must ensure that its +superclass subobject gets initialized. The typical way to do so is +through the use of superclass delegation: + + class A { + var x: Int + + init(x: Int) { + self.x = x + } + } + + class B : A { + var value: String + + init() { + value = "Hello" + super.init(5) // superclass delegation + } + } + +Swift implements two-phase initialization, which requires that all of +the instance variables of the subclass be initialized (either within the +class or within the initializer) before delegating to the superclass +initializer with `super.init`. + +If the superclass is a Swift class, superclass delegation is a direct +call to the named initializer in the superclass. If the superclass is an +Objective-C class, superclass delegation uses dynamic dispatch via +`objc_msgSendSuper` (and its variants). + +Peer Delegation +--------------- + +An initializer can delegate to one of its peer initializers, which then +takes responsibility for initializing this subobject and any superclass +subobjects: + + extension A { + init fromString(s: String) { + self.init(Int(s)) // peer delegation to init(Int) + } + } + +One cannot access any of the instance variables of `self` nor invoke any +instance methods on `self` before delegating to the peer initializer, +because the object has not yet been constructed. Additionally, one +cannot initialize the instance variables of `self`, prior to delegating +to the peer, because doing so would lead to double initializations. + +Peer delegation is always a direct call to the named initializer, and +always calls an initializer defined for the same type as the delegating +initializer. Despite the syntactic similarities, this is very different +from Objective-C's `[self init...]`, which can call methods in either +the superclass or subclass. + +Peer delegation is primarily useful when providing convenience +initializers without having to duplicate initialization code. However, +peer delegation is also the only viable way to implement an initializer +for a type within an extension that resides in a different resilience +domain than the definition of the type itself. For example, consider the +following extension of `A`: + + extension A { + init(i: Int, j: Int) { + x = i + j // initialize x + } + } + +If this extension is in a different resilience domain than the +definition of `A`, there is no way to ensure that this initializer is +initializing all of the instance variables of `A`: new instance +variables could be added to `A` in a future version (these would not be +properly initialized) and existing instance variables could become +computed properties (these would be initialized when they shouldn't be). + +Initializer Inheritance +----------------------- + +Initializers are *not* inherited by default. Each subclass takes +responsibility for its own initialization by declaring the initializers +one can use to create it. To make a superclass's initializer available +in a subclass, one can re-declare the initializer and then use +superclass delegation to call it: + + class C : A { + var value = "Hello" + + init(x: Int) { + super.init(x) // superclass delegation + } + } + +Although `C`'s initializer has the same parameters as `A`'s initializer, +it does not "override" `A`'s initializer because there is no dynamic +dispatch for initializers (but see below). + +We could syntax-optimize initializer inheritance if this becomes +onerous. DaveA provides a reasonable suggestion: + + class C : A { + var value = "Hello" + + @inherit init(Int) + } + +*Note*: one can only inherit an initializer into a class `C` if all of +the instance variables in that class have in-class initializers. + +Virtual Initializers +-------------------- + +The initializer model above only safely permits initialization when we +statically know the type of the complete object being initialized. For +example, this permits the construction `A(5)` but not the following: + + func createAnA(aClass: A.metatype) -> A { + return aClass(5) // error: no complete initializer accepting an ``Int`` + } + +The issue here is that, while `A` has an initializer accepting an `Int`, +it's not guaranteed that an arbitrary subclass of `A` will have such an +initializer. Even if we had that guarantee, there wouldn't necessarily +be any way to call the initializer, because (as noted above), there is +no dynamic dispatch for initializers. + +This is an unacceptable limitation for a few reasons. The most obvious +reason is that `NSCoding` depends on dynamic dispatch to +`-initWithCoder:` to deserialize an object of a class type that is +dynamically determined, and Swift classes must safely support this +paradigm. To address this limitation, we can add the `virtual` attribute +to turn an initializer into a virtual initializer: + + class A { + @virtual init(x: Int) { ... } + } + +Virtual initializers can be invoked when constructing an object using an +arbitrary value of metatype type (as in the `createAnA` example above), +using dynamic dispatch. Therefore, we need to ensure that a virtual +initializer is always a complete object initializer, which requires that +every subclass provide a definition for each virtual initializer defined +in its superclass. For example, the following class definition would be +ill-formed: + + class D : A { + var floating: Double + } + +because `D` does not provide an initializer accepting an `Int`. To +address this issue, one would add: + + class D : A { + var floating: Double + + @virtual init(x: Int) { + floating = 3.14159 + super.init(x) + } + } + +As a convenience, the compiler could synthesize virtual initializer +definitions when all of the instance variables in the subclass have +in-class initializers: + + class D2 : A { + var floating = 3.14159 + + /* compiler-synthesized */ + @virtual init(x: Int) { + super.init(x) + } + } + +This looks a lot like inherited initializers, and can eliminate some +boilerplate for simple subclasses. The primary downside is that the +synthesized implementation might not be the right one, e.g., it will +almost surely be wrong for an inherited `-initWithCoder:`. I don't think +this is worth doing. + +*Note*: as a somewhat unfortunate side effect of the terminology, the +initializers for structs and enums are considered to be virtual, because +they are guaranteed to be complete object initializers. If this bothers +us, we could use the term (and attribute) "complete" instead of +"virtual". I'd prefer to stick with "virtual" and accept the corner +case. + +Initializers in Protocols +------------------------- + +We currently ban initializers in protocols because we didn't have an +implementation model for them. Protocols, whether used via generics or +via existentials, use dynamic dispatch through the witness table. More +importantly, one of the important aspects of protocols is that, when a +given class `A` conforms to a protocol `P`, all of the subclasses of `A` +also conform to `P`. This property interacts directly with initializers: + + protocol NSCoding { + init withCoder(coder: NSCoder) + } + + class A : NSCoding { + init withCoder(coder: NSCoder) { /* ... */ } + } + + class B : A { + // conforms to NSCoding? + } + +Here, `A` appears to conform to `NSCoding` because it provides a +matching initializer. `B` should conform to `NSCoding`, because it +should inherit its conformance from `A`, but the lack of an +`initWithCoder:` initializer causes problems. The fix here is to require +that the witness be a virtual initializer, which guarantees that all of +the subclasses will have the same initializer. Thus, the definition of +`A` above will be ill-formed unless `initWithCoder:` is made virtual: + + protocol NSCoding { + init withCoder(coder: NSCoder) + } + + class A : NSCoding { + @virtual init withCoder(coder: NSCoder) { /* ... */ } + } + + class B : A { + // either error (due to missing initWithCoder) or synthesized initWithCoder: + } + +As noted earlier, the initializers of structs and enums are considered +virtual. + +Objective-C Interoperability +---------------------------- + +The initialization model described above guarantees that objects are +properly initialized before they are used, covering all of the major use +cases for initialization while maintaining soundness. Objective-C has a +very different initialization model with which Swift must interoperate. + +### Objective-C Entrypoints + +Each Swift initializer definition produces a corresponding Objective-C +init method. The existence of this init method allows object +construction from Objective-C (both directly via `[[A alloc] init:5]` +and indirectly via, e.g., `[obj initWithCoder:coder]`) and +initialization of the superclass subobject when an Objective-C class +inherits from a Swift class (e.g., `[super initWithCoder:coder]`). + +Note that, while Swift's initializers are not inherited and cannot +override, this is only true *in Swift code*. If a subclass defines an +initializer with the same Objective-C selector as an initializer in its +superclass, the Objective-C init method produced for the former will +override the Objective-C init method produced for the latter. + +### Objective-C Restrictions + +The emission of Objective-C init methods for Swift initializers open up +a few soundness problems, illustrated here: + + @interface A + @end + + @implementation A + - init { + return [self initWithInt:5]; + } + + - initWithInt:(int)x { + // initialize me + } + + - initWithString:(NSString *)s { + // initialize me + } + @end + + class B1 : A { + var dict: NSDictionary + + init withInt(x: Int) { + dict = [] + super.init() // loops forever, initializing dict repeatedly + } + } + + class B2 : A { + } + + @interface C : B2 + @end + + @implementation C + @end + + void getCFromString(NSString *str) { + return [C initWithString:str]; // doesn't initialize B's dict ivar + } + +The first problem, with `B1`, comes from `A`'s dispatched delegation to +`-initWithInt:`, which is overridden by `B1`'s initializer with the same +selector. We can address this problem by enforcing that superclass +delegation to an Objective-C superclass initializer refer to a +designated initializer of that superclass when that class has at least +one initializer marked as a designated initializer. + +The second problem, with `C`, comes from Objective-C's implicit +inheritance of initializers. We can address this problem by specifying +that init methods in Objective-C are never visible through Swift +classes, making the message send `[C initWithString:str]` ill-formed. +This is a relatively small Clang-side change. + +### Remaining Soundness Holes + +Neither of the above "fixes" are complete. The first depends entirely on +the adoption of a not-yet-implemented Clang attribute to mark the +designated initializers for Objective-C classes, while the second is +(almost trivially) defeated by passing the `-initWithString:` message to +an object of type `id` or using some other dynamic reflection. + +If we want to close these holes tighter, we could stop emitting +Objective-C init methods for Swift initializers. Instead, we would fake +the init method declarations when importing Swift modules into Clang, +and teach Clang's CodeGen to emit calls directly to the Swift +initializers. It would still not be perfect (e.g., some variant of the +problem with `C` would persist), but it would be closer. I suspect that +this is far more work than it is worth, and that the "fixes" described +above are sufficient. diff --git a/docs/proposals/Initialization.rst b/docs/proposals/Initialization.rst deleted file mode 100644 index 376bdd4a06fe7..0000000000000 --- a/docs/proposals/Initialization.rst +++ /dev/null @@ -1,341 +0,0 @@ -:orphan: - -Initialization -============== - -.. contents:: - -Superclass Delegation ---------------------- -The initializer for a class that has a superclass must ensure that its -superclass subobject gets initialized. The typical way to do so is -through the use of superclass delegation:: - - class A { - var x: Int - - init(x: Int) { - self.x = x - } - } - - class B : A { - var value: String - - init() { - value = "Hello" - super.init(5) // superclass delegation - } - } - -Swift implements two-phase initialization, which requires that all of -the instance variables of the subclass be initialized (either within -the class or within the initializer) before delegating to the -superclass initializer with ``super.init``. - -If the superclass is a Swift class, superclass delegation is a direct -call to the named initializer in the superclass. If the superclass is -an Objective-C class, superclass delegation uses dynamic dispatch via -``objc_msgSendSuper`` (and its variants). - -Peer Delegation ---------------- -An initializer can delegate to one of its peer initializers, which -then takes responsibility for initializing this subobject and any -superclass subobjects:: - - extension A { - init fromString(s: String) { - self.init(Int(s)) // peer delegation to init(Int) - } - } - -One cannot access any of the instance variables of ``self`` nor invoke -any instance methods on ``self`` before delegating to the peer -initializer, because the object has not yet been -constructed. Additionally, one cannot initialize the instance -variables of ``self``, prior to delegating to the peer, because doing -so would lead to double initializations. - -Peer delegation is always a direct call to the named initializer, and -always calls an initializer defined for the same type as the -delegating initializer. Despite the syntactic similarities, this is -very different from Objective-C's ``[self init...]``, which can call -methods in either the superclass or subclass. - -Peer delegation is primarily useful when providing convenience -initializers without having to duplicate initialization code. However, -peer delegation is also the only viable way to implement an -initializer for a type within an extension that resides in a different -resilience domain than the definition of the type itself. For example, -consider the following extension of ``A``:: - - extension A { - init(i: Int, j: Int) { - x = i + j // initialize x - } - } - -If this extension is in a different resilience domain than the -definition of ``A``, there is no way to ensure that this initializer -is initializing all of the instance variables of ``A``: new instance -variables could be added to ``A`` in a future version (these would not -be properly initialized) and existing instance variables could become -computed properties (these would be initialized when they shouldn't -be). - -Initializer Inheritance ------------------------ -Initializers are *not* inherited by default. Each subclass takes -responsibility for its own initialization by declaring the -initializers one can use to create it. To make a superclass's -initializer available in a subclass, one can re-declare the -initializer and then use superclass delegation to call it:: - - class C : A { - var value = "Hello" - - init(x: Int) { - super.init(x) // superclass delegation - } - } - -Although ``C``'s initializer has the same parameters as ``A``'s -initializer, it does not "override" ``A``'s initializer because there -is no dynamic dispatch for initializers (but see below). - -We could syntax-optimize initializer inheritance if this becomes -onerous. DaveA provides a reasonable suggestion:: - - class C : A { - var value = "Hello" - - @inherit init(Int) - } - -*Note*: one can only inherit an initializer into a class ``C`` if all -of the instance variables in that class have in-class initializers. - -Virtual Initializers --------------------- -The initializer model above only safely permits initialization when we -statically know the type of the complete object being initialized. For -example, this permits the construction ``A(5)`` but not the -following:: - - func createAnA(aClass: A.metatype) -> A { - return aClass(5) // error: no complete initializer accepting an ``Int`` - } - -The issue here is that, while ``A`` has an initializer accepting an -``Int``, it's not guaranteed that an arbitrary subclass of ``A`` will -have such an initializer. Even if we had that guarantee, there -wouldn't necessarily be any way to call the initializer, because (as -noted above), there is no dynamic dispatch for initializers. - -This is an unacceptable limitation for a few reasons. The most obvious -reason is that ``NSCoding`` depends on dynamic dispatch to -``-initWithCoder:`` to deserialize an object of a class type that is -dynamically determined, and Swift classes must safely support this -paradigm. To address this limitation, we can add the ``virtual`` -attribute to turn an initializer into a virtual initializer:: - - class A { - @virtual init(x: Int) { ... } - } - -Virtual initializers can be invoked when constructing an object using -an arbitrary value of metatype type (as in the ``createAnA`` example -above), using dynamic dispatch. Therefore, we need to ensure that a -virtual initializer is always a complete object initializer, which -requires that every subclass provide a definition for each virtual -initializer defined in its superclass. For example, the following -class definition would be ill-formed:: - - class D : A { - var floating: Double - } - -because ``D`` does not provide an initializer accepting an ``Int``. To -address this issue, one would add:: - - class D : A { - var floating: Double - - @virtual init(x: Int) { - floating = 3.14159 - super.init(x) - } - } - -As a convenience, the compiler could synthesize virtual initializer -definitions when all of the instance variables in the subclass have -in-class initializers:: - - class D2 : A { - var floating = 3.14159 - - /* compiler-synthesized */ - @virtual init(x: Int) { - super.init(x) - } - } - -This looks a lot like inherited initializers, and can eliminate some -boilerplate for simple subclasses. The primary downside is that the -synthesized implementation might not be the right one, e.g., it will -almost surely be wrong for an inherited ``-initWithCoder:``. I don't -think this is worth doing. - -*Note*: as a somewhat unfortunate side effect of the terminology, the -initializers for structs and enums are considered to be virtual, -because they are guaranteed to be complete object initializers. If -this bothers us, we could use the term (and attribute) "complete" -instead of "virtual". I'd prefer to stick with "virtual" and accept -the corner case. - -Initializers in Protocols -------------------------- -We currently ban initializers in protocols because we didn't have an -implementation model for them. Protocols, whether used via generics or -via existentials, use dynamic dispatch through the witness table. More -importantly, one of the important aspects of protocols is that, when a -given class ``A`` conforms to a protocol ``P``, all of the subclasses -of ``A`` also conform to ``P``. This property interacts directly with -initializers:: - - protocol NSCoding { - init withCoder(coder: NSCoder) - } - - class A : NSCoding { - init withCoder(coder: NSCoder) { /* ... */ } - } - - class B : A { - // conforms to NSCoding? - } - -Here, ``A`` appears to conform to ``NSCoding`` because it provides a -matching initializer. ``B`` should conform to ``NSCoding``, because it -should inherit its conformance from ``A``, but the lack of an -``initWithCoder:`` initializer causes problems. The fix here is to -require that the witness be a virtual initializer, which guarantees -that all of the subclasses will have the same initializer. Thus, the -definition of ``A`` above will be ill-formed unless ``initWithCoder:`` -is made virtual:: - - protocol NSCoding { - init withCoder(coder: NSCoder) - } - - class A : NSCoding { - @virtual init withCoder(coder: NSCoder) { /* ... */ } - } - - class B : A { - // either error (due to missing initWithCoder) or synthesized initWithCoder: - } - -As noted earlier, the initializers of structs and enums are considered -virtual. - -Objective-C Interoperability ----------------------------- -The initialization model described above guarantees that objects are -properly initialized before they are used, covering all of the major -use cases for initialization while maintaining soundness. Objective-C -has a very different initialization model with which Swift must -interoperate. - -Objective-C Entrypoints -~~~~~~~~~~~~~~~~~~~~~~~ -Each Swift initializer definition produces a corresponding Objective-C -init method. The existence of this init method allows object -construction from Objective-C (both directly via ``[[A alloc] -init:5]`` and indirectly via, e.g., ``[obj initWithCoder:coder]``) -and initialization of the superclass subobject when an Objective-C class -inherits from a Swift class (e.g., ``[super initWithCoder:coder]``). - -Note that, while Swift's initializers are not inherited and cannot -override, this is only true *in Swift code*. If a subclass defines an -initializer with the same Objective-C selector as an initializer in -its superclass, the Objective-C init method produced for the former -will override the Objective-C init method produced for the -latter. - -Objective-C Restrictions -~~~~~~~~~~~~~~~~~~~~~~~~ -The emission of Objective-C init methods for Swift initializers open -up a few soundness problems, illustrated here:: - - @interface A - @end - - @implementation A - - init { - return [self initWithInt:5]; - } - - - initWithInt:(int)x { - // initialize me - } - - - initWithString:(NSString *)s { - // initialize me - } - @end - - class B1 : A { - var dict: NSDictionary - - init withInt(x: Int) { - dict = [] - super.init() // loops forever, initializing dict repeatedly - } - } - - class B2 : A { - } - - @interface C : B2 - @end - - @implementation C - @end - - void getCFromString(NSString *str) { - return [C initWithString:str]; // doesn't initialize B's dict ivar - } - -The first problem, with ``B1``, comes from ``A``'s dispatched -delegation to ``-initWithInt:``, which is overridden by ``B1``'s -initializer with the same selector. We can address this problem by -enforcing that superclass delegation to an Objective-C superclass -initializer refer to a designated initializer of that superclass when -that class has at least one initializer marked as a designated -initializer. - -The second problem, with ``C``, comes from Objective-C's implicit -inheritance of initializers. We can address this problem by specifying -that init methods in Objective-C are never visible through Swift -classes, making the message send ``[C initWithString:str]`` -ill-formed. This is a relatively small Clang-side change. - -Remaining Soundness Holes -~~~~~~~~~~~~~~~~~~~~~~~~~ -Neither of the above "fixes" are complete. The first depends entirely -on the adoption of a not-yet-implemented Clang attribute to mark the -designated initializers for Objective-C classes, while the second is -(almost trivially) defeated by passing the ``-initWithString:`` -message to an object of type ``id`` or using some other dynamic -reflection. - -If we want to close these holes tighter, we could stop emitting -Objective-C init methods for Swift initializers. Instead, we would -fake the init method declarations when importing Swift modules into -Clang, and teach Clang's CodeGen to emit calls directly to the Swift -initializers. It would still not be perfect (e.g., some variant of the -problem with ``C`` would persist), but it would be closer. I suspect -that this is far more work than it is worth, and that the "fixes" -described above are sufficient. diff --git a/docs/proposals/InitializerInheritance.md b/docs/proposals/InitializerInheritance.md new file mode 100644 index 0000000000000..1a111eda804ee --- /dev/null +++ b/docs/proposals/InitializerInheritance.md @@ -0,0 +1,387 @@ +Initializer Inheritance +======================= + +Authors + +: Doug Gregor, John McCall + +Introduction +------------ + +This proposal introduces the notion of initializer inheritance into the +Swift initialization model. The intent is to more closely model +Objective-C's initializer inheritance model while maintaining memory +safety. + +Background +---------- + +An initializer is a definition, and it belongs to the class that defines +it. However, it also has a signature, and it makes sense to talk about +other initializers in related classes that share that signature. + +Initializers come in two basic kinds: complete object and subobject. +That is, an initializer either takes responsibility for initializing the +complete object, potentially including derived class subobjects, or it +takes responsibility for initializing only the subobjects of its class +and its superclasses. + +There are three kinds of delegation: + +- **super**: runs an initializer belonging to a superclass (in ObjC, + `[super init…]`; in Swift, `super.init(...)`). +- **peer**: runs an initializer belonging to the current class (ObjC + does not have syntax for this, although it is supported by the + runtime; in Swift, the current meaning of `self.init(...)`) +- **dispatched**: given a signature, runs the initializer with that + signature that is either defined or inherited by the most-derived + class (in ObjC, `[self init…]`; not currently supported by Swift) + +We can also distinguish two ways to originally invoke an initializer: + +- **direct**: the most-derived class is statically known +- **indirect**: the most-derived class is not statically known and an + initialization must be dispatched + +Either kind of dispatched initialization poses a soundness problem +because there may not be a sound initializer with any given signature in +the most-derived class. In ObjC, initializers are normal instance +methods and are therefore inherited like normal, but this isn’t really +quite right; initialization is different from a normal method in that +it’s not inherently sensible to require subclasses to provide +initializers at all the signatures that their superclasses provide. + +### Subobject initializers + +The defining class of a subobject initializer is central to its +behavior. It can be soundly inherited by a class C only if is trivial to +initialize the ivars of C, but it’s convenient to ignore that and assume +that subobjects will always trivially wrap and delegate to superclass +subobject initializers. + +A subobject initializer must either (1) delegate to a peer subobject +initializer or (2) take responsibility for initializing all ivars of its +defining class and delegate to a subobject initializer of its +superclass. It cannot initialize any ivars of its defining class if it +then delegates to a peer subobject initializer, although it can +re-assign ivars after initialization completes. + +A subobject initializer soundly acts like a complete object initializer +of a class C if and only if it is defined by C. + +### Complete object initializers + +The defining class of a complete object initializer doesn’t really +matter. In principle, complete object initializers could just as well be +freestanding functions to which a metatype is passed. It can make sense +to inherit a complete object initializer. + +A complete object initializer must either (1) delegate (in any way) to +another complete object initializer or (2) delegate (dispatched) to a +subobject initializer of the most-derived class. It may not initialize +any ivars, although it can re-assign them after initialization +completes. + +A complete object initializer soundly acts like a complete object +initializer of a class C if and only if it delegates to an initializer +which soundly acts like a complete object initializer of C. + +These rules are not obvious to check statically because they’re +dependent on the dynamic value of the most-derived class C. Therefore +any ability to check them depends on restricting C somehow relative to +the defining class of the initializer. Since, statically, we only know +the defining class of the initializer, we can’t establish soundness +solely at the definition site; instead we have to prevent unsound +initializers from being called. + +### Guaranteed initializers + +The chief soundness question comes back to dispatched initialization: +when can we reasonably assume that the most-derived class provides an +initializer with a given signature? + +Only a complete object initializer can perform dispatched delegation. A +dispatched delegation which invokes another complete object initializer +poses no direct soundness issues. The dynamic requirement for soundness +is that, eventually, the chain of dispatches leads to a subobject +initializer that soundly acts like a complete object initializer of the +most-derived class. + +### Virtual initializers + +The above condition is not sufficient to make indirect initialization +sound, because it relies on the ability to simply not use an initializer +in cases where its delegation behavior isn't known to be sound, and we +can’t do that to arbitrary code. For that, we would need true virtual +initializers. + +A virtual initializer is a contract much more like that of a standard +virtual method: it guarantees that every subclass will either define or +inherit a constructor with a given signature, which is easy to check at +the time of definition of the subclass. + +Proposal +-------- + +Currently, all Swift initializers are subobject initializers, and there +is no way to express the notion of a complete subobject initializer. We +propose to introduce complete subobject initializers into Swift and to +make them inheritable when we can guarantee that doing so is safe. + +### Complete object initializers + +Introduce the notion of a complete object initializer, which is written +as an initializer with `Self` as its return type \[\#\], e.g.: + + init() -> Self { + // ... + } + +The use of `Self` here fits well with dynamic `Self`, because a complete +object initializer returns an instance of the dynamic type being +initialized (rather than the type that defines the initializer). + +A complete object initializer must delegate to another initializer via +`self.init`, which may itself be either a subobject initializer or a +complete object initializer. The delegation itself is dispatched. For +example: + + class A { + var title: String + + init() -> Self { // complete object initializer + self.init(withTitle:"The Next Great American Novel") + } + + init withTitle(title: String) { // subobject initializer + self.title = title + } + } + +Subobject initializers become more restricted. They must initialize A's +instance variables and then perform super delegation to a subobject +initializer of the superclass (if any). + +### Initializer inheritance + +A class inherits the complete object initializers of its direct +superclass when it overrides all of the subobject initializers of its +direct superclass. Subobject initializers are never inherited. Some +examples: + + class B1 : A { + var counter: Int + + init withTitle(title: String) { // subobject initializer + counter = 0 + super.init(withTitle:title) + } + + // inherits A's init() + } + + class B2 : A { + var counter: Int + + init withTitle(title: String) -> Self { // complete object initializer + self.init(withTitle: title, initialCount: 0) + } + + init withTitle(title: String) initialCount(Int) { // subobject initializer + counter = initialCount + super.init(withTitle:title) + } + + // inherits A's init() + } + + class B3 : A { + var counter: Int + + init withInitialCount(initialCount: Int) { // subobject initializer + counter = initialCount + super.init(withTitle: "Unnamed") + } + + init withStringCount(str: String) -> Self { // complete object initializer + var initialCount = 0 + if let count = str.toInt() { initialCount = count } + self.init(withInitialCount: initialCount) + } + + // does not inherit A's init(), because init withTitle(String) is not + // overridden. + } + +`B3` does not override `A`'s subobject initializer, so it does not +inherit `init()`. Classes `B1` and `B2`, however, both inherit the +initializer `init()` from `A`, because both override its only subobject +initializer, `init withTitle(String)`. This means that one can construct +either a `B1` or a `B2` with no arguments: + + B1() // okay + B2() // okay + B3() // error + +That `B1` uses a subobject initializer to override it's superclass's +subobject initializer while `B2` uses a complete object initializer has +an effect on future subclasses. A few more examples: + + class C1 : B1 { + init withTitle(title: String) { // subobject initializer + super.init(withTitle:title) + } + + init withTitle(title: String) initialCount(Int) { // subobject initializer + counter = initialCount + super.init(withTitle:title) + } + } + + class C2 : B2 { + init withTitle(title: String) initialCount(Int) { // subobject initializer + super.init(withTitle: title, initialCount:initialCount) + } + + // inherits A's init(), B2's init withTitle(String) + } + + class C3 : B3 { + init withInitialCount(initialCount: Int) { // subobject initializer + super.init(withInitialCount: initialCount) + } + + // inherits B3's init withStringCount(str: String) + // does not inherit A's init() + } + +### Virtual initializers + +With the initializer inheritance rules described above, there is no +guarantee that one can dynamically dispatch to an initializer via a +metatype of the class. For example: + + class D { + init() { } + } + + func f(meta: D.Type) { + meta() // error: no guarantee that an arbitrary of subclass D has an init() + } + +Virtual initializers, which are initializers that have the `virtual` +attribute, are guaranteed to be available in every subclass of `D`. For +example, if `D` was written as: + + class D { + @virtual init() { } + } + + func f(meta: D.Type) { + meta() // okay: every sublass of D guaranteed to have an init() + } + +Note that `@virtual` places a requirement on all subclasses to ensure +that an initializer with the same signature is available in every +subclass. For example: + + class E1 : D { + var title: String + + // error: E1 must provide init() + } + + class E2 : D { + var title: String + + @virtual init() { + title = "Unnamed" + super.init() + } + + // okay, init() is available here + } + + class E3 : D { + var title: String + + @virtual init() -> Self { + self.init(withTitle: "Unnamed") + } + + init withTitle(title: String) { + self.title = title + super.init() + } + } + +Whether an initializer is virtual is orthogonal to whether it is a +complete object or subobject initializer. However, an inherited complete +object initializer can be used to satisfy the requirement for a virtual +requirement. For example, `E3`'s subclasses need not provide an `init()` +if they override `init withTitle(String)`: + + class F3A : E3 { + init withTitle(title: String) { + super.init(withTitle: title) + } + + // okay: inherited ``init()`` from E3 satisfies requirement for virtual init() + } + + class F3B : E3 { + // error: requirement for virtual init() not satisfied, because it is neither defined nor inherited + } + + class F3C : E3 { + @virtual init() { + super.init(withTitle: "TSPL") + } + + // okay: satisfies requirement for virtual init(). + } + +### Objective-C interoperability + +When an Objective-C class that contains at least one +designated-initializer annotation (i.e., via +`NS_DESIGNATED_INITIALIZER`) is imported into Swift, it's designated +initializers are considered subobject initializers. Any non-designed +initializers (i.e., secondary or convenience initializers) are +considered to be complete object initializers. No other special-case +behavior is warranted here. + +When an Objective-C class with no designated-initializer annotations is +imported into Swift, all initializers in the same module as the class +definition are subobject initializers, while initializers in a different +module are complete object initializers. This effectively means that +subclassing Objective-C classes without designated-initializer +annotations will provide little or no initializer inheritance, because +one would have to override nearly *all* of its initializers before +getting the others inherited. This seems acceptable so long as we get +designated-initializer annotations into enough of the SDK. + +In Objective-C, initializers are always inherited, so an error of +omission on the Swift side (failing to override a subobject initializer +from a superclass) can result in runtime errors if an Objective-C +framework messages that initializer. For example, consider a trivial +`NSDocument`: + + class MyDocument : NSDocument { + var title: String + } + +In Swift, there would be no way to create an object of type +`MyDocument`. However, the frameworks will allocate an instance of +`MyDocument` and then send an message such as +`initWithContentsOfURL:ofType:error:` to the object. This will find +`-[NSDocument initWithContentsOfURL:ofType:error:]`, which delegates to +`-[NSDocument init]`, leaving `MyDocument`'s stored properties +uninitialized. + +We can improve the experience slightly by producing a diagnostic when +there are no initializers for a given class. However, a more +comprehensive approach is to emit Objective-C entry points for each of +the subobject initializers of the direct superclass that have not been +implemented. These entry points would immediately abort with some +diagnostic indicating that the initializer needs to be implemented. diff --git a/docs/proposals/InitializerInheritance.rst b/docs/proposals/InitializerInheritance.rst deleted file mode 100644 index 6fbb8118deb13..0000000000000 --- a/docs/proposals/InitializerInheritance.rst +++ /dev/null @@ -1,396 +0,0 @@ -:orphan: - -Initializer Inheritance -======================= - -:Authors: Doug Gregor, John McCall - -.. contents:: - -Introduction ------------- -This proposal introduces the notion of initializer inheritance into -the Swift initialization model. The intent is to more closely model -Objective-C's initializer inheritance model while maintaining memory -safety. - -Background ----------- -An initializer is a definition, and it belongs to the class that -defines it. However, it also has a signature, and it makes sense to -talk about other initializers in related classes that share that -signature. - -Initializers come in two basic kinds: complete object and subobject. -That is, an initializer either takes responsibility for initializing -the complete object, potentially including derived class subobjects, -or it takes responsibility for initializing only the subobjects of its -class and its superclasses. - -There are three kinds of delegation: - -* **super**: runs an initializer belonging to a superclass (in ObjC, - ``[super init…]``; in Swift, ``super.init(...)``). - -* **peer**: runs an initializer belonging to the current class (ObjC - does not have syntax for this, although it is supported by the - runtime; in Swift, the current meaning of ``self.init(...)``) - -* **dispatched**: given a signature, runs the initializer with that - signature that is either defined or inherited by the most-derived - class (in ObjC, ``[self init…]``; not currently supported by Swift) - -We can also distinguish two ways to originally invoke an initializer: - -* **direct**: the most-derived class is statically known - -* **indirect**: the most-derived class is not statically known and - an initialization must be dispatched - -Either kind of dispatched initialization poses a soundness problem -because there may not be a sound initializer with any given signature -in the most-derived class. In ObjC, initializers are normal instance -methods and are therefore inherited like normal, but this isn’t really -quite right; initialization is different from a normal method in that -it’s not inherently sensible to require subclasses to provide -initializers at all the signatures that their superclasses provide. - -Subobject initializers -~~~~~~~~~~~~~~~~~~~~~~ -The defining class of a subobject initializer is central to its -behavior. It can be soundly inherited by a class C only if is trivial -to initialize the ivars of C, but it’s convenient to ignore that and -assume that subobjects will always trivially wrap and delegate to -superclass subobject initializers. - -A subobject initializer must either (1) delegate to a peer subobject -initializer or (2) take responsibility for initializing all ivars of -its defining class and delegate to a subobject initializer of its -superclass. It cannot initialize any ivars of its defining class if -it then delegates to a peer subobject initializer, although it can -re-assign ivars after initialization completes. - -A subobject initializer soundly acts like a complete object -initializer of a class C if and only if it is defined by C. - -Complete object initializers -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The defining class of a complete object initializer doesn’t really -matter. In principle, complete object initializers could just as well -be freestanding functions to which a metatype is passed. It can make -sense to inherit a complete object initializer. - -A complete object initializer must either (1) delegate (in any way) to -another complete object initializer or (2) delegate (dispatched) to a -subobject initializer of the most-derived class. It may not -initialize any ivars, although it can re-assign them after -initialization completes. - -A complete object initializer soundly acts like a complete object -initializer of a class C if and only if it delegates to an initializer -which soundly acts like a complete object initializer of C. - -These rules are not obvious to check statically because they’re -dependent on the dynamic value of the most-derived class C. Therefore -any ability to check them depends on restricting C somehow relative to -the defining class of the initializer. Since, statically, we only -know the defining class of the initializer, we can’t establish -soundness solely at the definition site; instead we have to prevent -unsound initializers from being called. - -Guaranteed initializers -~~~~~~~~~~~~~~~~~~~~~~~ -The chief soundness question comes back to dispatched initialization: -when can we reasonably assume that the most-derived class provides an -initializer with a given signature? - -Only a complete object initializer can perform dispatched delegation. -A dispatched delegation which invokes another complete object -initializer poses no direct soundness issues. The dynamic requirement -for soundness is that, eventually, the chain of dispatches leads to a -subobject initializer that soundly acts like a complete object -initializer of the most-derived class. - -Virtual initializers -~~~~~~~~~~~~~~~~~~~~ -The above condition is not sufficient to make indirect initialization -sound, because it relies on the ability to simply not use an -initializer in cases where its delegation behavior isn't known to be -sound, and we can’t do that to arbitrary code. For that, we would -need true virtual initializers. - -A virtual initializer is a contract much more like that of a standard -virtual method: it guarantees that every subclass will either define -or inherit a constructor with a given signature, which is easy to -check at the time of definition of the subclass. - -Proposal --------- -Currently, all Swift initializers are subobject initializers, and -there is no way to express the notion of a complete subobject -initializer. We propose to introduce complete subobject initializers -into Swift and to make them inheritable when we can guarantee that -doing so is safe. - -Complete object initializers -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -Introduce the notion of a complete object initializer, which is -written as an initializer with ``Self`` as its return type [#], e.g.:: - - init() -> Self { - // ... - } - -The use of ``Self`` here fits well with dynamic ``Self``, because a -complete object initializer returns an instance of the dynamic type -being initialized (rather than the type that defines the initializer). - -A complete object initializer must delegate to another initializer via -``self.init``, which may itself be either a subobject initializer or a -complete object initializer. The delegation itself is dispatched. For -example:: - - class A { - var title: String - - init() -> Self { // complete object initializer - self.init(withTitle:"The Next Great American Novel") - } - - init withTitle(title: String) { // subobject initializer - self.title = title - } - } - -Subobject initializers become more restricted. They must initialize -A's instance variables and then perform super delegation to a -subobject initializer of the superclass (if any). - -Initializer inheritance -~~~~~~~~~~~~~~~~~~~~~~~ - -A class inherits the complete object initializers of its direct -superclass when it overrides all of the subobject initializers of its -direct superclass. Subobject initializers are never inherited. Some -examples:: - - class B1 : A { - var counter: Int - - init withTitle(title: String) { // subobject initializer - counter = 0 - super.init(withTitle:title) - } - - // inherits A's init() - } - - class B2 : A { - var counter: Int - - init withTitle(title: String) -> Self { // complete object initializer - self.init(withTitle: title, initialCount: 0) - } - - init withTitle(title: String) initialCount(Int) { // subobject initializer - counter = initialCount - super.init(withTitle:title) - } - - // inherits A's init() - } - - class B3 : A { - var counter: Int - - init withInitialCount(initialCount: Int) { // subobject initializer - counter = initialCount - super.init(withTitle: "Unnamed") - } - - init withStringCount(str: String) -> Self { // complete object initializer - var initialCount = 0 - if let count = str.toInt() { initialCount = count } - self.init(withInitialCount: initialCount) - } - - // does not inherit A's init(), because init withTitle(String) is not - // overridden. - } - -``B3`` does not override ``A``'s subobject initializer, so it does not -inherit ``init()``. Classes ``B1`` and ``B2``, however, both inherit -the initializer ``init()`` from ``A``, because both override its only -subobject initializer, ``init withTitle(String)``. This means that one -can construct either a ``B1`` or a ``B2`` with no arguments:: - - B1() // okay - B2() // okay - B3() // error - -That ``B1`` uses a subobject initializer to override it's superclass's -subobject initializer while ``B2`` uses a complete object initializer -has an effect on future subclasses. A few more examples:: - - class C1 : B1 { - init withTitle(title: String) { // subobject initializer - super.init(withTitle:title) - } - - init withTitle(title: String) initialCount(Int) { // subobject initializer - counter = initialCount - super.init(withTitle:title) - } - } - - class C2 : B2 { - init withTitle(title: String) initialCount(Int) { // subobject initializer - super.init(withTitle: title, initialCount:initialCount) - } - - // inherits A's init(), B2's init withTitle(String) - } - - class C3 : B3 { - init withInitialCount(initialCount: Int) { // subobject initializer - super.init(withInitialCount: initialCount) - } - - // inherits B3's init withStringCount(str: String) - // does not inherit A's init() - } - -Virtual initializers -~~~~~~~~~~~~~~~~~~~~ -With the initializer inheritance rules described above, there is no -guarantee that one can dynamically dispatch to an initializer via a -metatype of the class. For example:: - - class D { - init() { } - } - - func f(meta: D.Type) { - meta() // error: no guarantee that an arbitrary of subclass D has an init() - } - -Virtual initializers, which are initializers that have the ``virtual`` -attribute, are guaranteed to be available in every subclass of -``D``. For example, if ``D`` was written as:: - - class D { - @virtual init() { } - } - - func f(meta: D.Type) { - meta() // okay: every sublass of D guaranteed to have an init() - } - -Note that ``@virtual`` places a requirement on all subclasses to -ensure that an initializer with the same signature is available in -every subclass. For example:: - - class E1 : D { - var title: String - - // error: E1 must provide init() - } - - class E2 : D { - var title: String - - @virtual init() { - title = "Unnamed" - super.init() - } - - // okay, init() is available here - } - - class E3 : D { - var title: String - - @virtual init() -> Self { - self.init(withTitle: "Unnamed") - } - - init withTitle(title: String) { - self.title = title - super.init() - } - } - -Whether an initializer is virtual is orthogonal to whether it is a -complete object or subobject initializer. However, an inherited -complete object initializer can be used to satisfy the requirement for -a virtual requirement. For example, ``E3``'s subclasses need not -provide an ``init()`` if they override ``init withTitle(String)``:: - - class F3A : E3 { - init withTitle(title: String) { - super.init(withTitle: title) - } - - // okay: inherited ``init()`` from E3 satisfies requirement for virtual init() - } - - class F3B : E3 { - // error: requirement for virtual init() not satisfied, because it is neither defined nor inherited - } - - class F3C : E3 { - @virtual init() { - super.init(withTitle: "TSPL") - } - - // okay: satisfies requirement for virtual init(). - } - -Objective-C interoperability -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -When an Objective-C class that contains at least one -designated-initializer annotation (i.e., via -``NS_DESIGNATED_INITIALIZER``) is imported into Swift, it's designated -initializers are considered subobject initializers. Any non-designed -initializers (i.e., secondary or convenience initializers) are -considered to be complete object initializers. No other special-case -behavior is warranted here. - -When an Objective-C class with no designated-initializer annotations -is imported into Swift, all initializers in the same module as the -class definition are subobject initializers, while initializers in a -different module are complete object initializers. This effectively -means that subclassing Objective-C classes without designated-initializer -annotations will provide little or no initializer inheritance, because -one would have to override nearly *all* of its initializers before -getting the others inherited. This seems acceptable so long as we get -designated-initializer annotations into enough of the SDK. - -In Objective-C, initializers are always inherited, so an error of -omission on the Swift side (failing to override a subobject -initializer from a superclass) can result in runtime errors if an -Objective-C framework messages that initializer. For example, consider -a trivial ``NSDocument``:: - - class MyDocument : NSDocument { - var title: String - } - -In Swift, there would be no way to create an object of type -``MyDocument``. However, the frameworks will allocate an instance of -``MyDocument`` and then send an message such as -``initWithContentsOfURL:ofType:error:`` to the object. This will find -``-[NSDocument initWithContentsOfURL:ofType:error:]``, which delegates -to ``-[NSDocument init]``, leaving ``MyDocument``'s stored properties -uninitialized. - -We can improve the experience slightly by producing a diagnostic when -there are no initializers for a given class. However, a more -comprehensive approach is to emit Objective-C entry points for each of -the subobject initializers of the direct superclass that have not been -implemented. These entry points would immediately abort with some -diagnostic indicating that the initializer needs to be -implemented. - -.. [#] Syntax suggestion from Joe Groff. - diff --git a/docs/proposals/InoutCOWOptimization.md b/docs/proposals/InoutCOWOptimization.md new file mode 100644 index 0000000000000..4d3e86f57be47 --- /dev/null +++ b/docs/proposals/InoutCOWOptimization.md @@ -0,0 +1,191 @@ +Copy-On-Write Optimization of `inout` Values +============================================ + +Authors + +: Dave Abrahams, Joe Groff + +Summary + +: Our writeback model interacts with Copy-On-Write (COW) to cause some + surprising inefficiencies, such as O(N) performance for + `x[0][0] = 1`. We propose a modified COW optimization that + recovers O(1) performance for these cases and supports the efficient + use of slices in algorithm implementation. + +Whence the Problem? +------------------- + +The problem is caused as follows: + +- COW depends on the programmer being able to mediate all writes (so + she can copy if necessary) +- Writes to container elements and slices are mediated through + subscript setters, so in : + + x[0].mutate() + + we “`subscript get`” `x[0]` into a temporary, mutate the temporary, + and “`subscript set`” it back into `x[0]`. + +- When the element itself is a COW type, that temporary implies a + retain count of at least 2 on the element's buffer. +- Therefore, mutating such an element causes an expensive copy, *even + when the element's buffer isn't otherwise shared*. + +Naturally, this problem generalizes to any COW value backed by a +getter/setter pair, such as a computed or resilient `String` property: + + anObject.title.append('.') // O(N) + +Interaction With Slices +----------------------- + +Consider the classic divide-and-conquer algorithm QuickSort, which could +be written as follows: + +The implicit `inout` on the target of the recursive `quickSort` calls +currently forces two allocations and O(N) copies in each layer of the +QuickSort implementation. Note that this problem applies to simple +containers such as `Int[]`, not just containers of COW elements. + +Without solving this problem, mutating algorithms must operate on +`MutableCollection`s and pairs of their `Index` types, and we must hope +the ARC optimizer is able to eliminate the additional reference at the +top-level call. However, that does nothing for the cases mentioned in +the previous section. + +Our Solution +------------ + +We need to prevent lvalues created in an `inout` context from forcing a +copy-on-write. To accomplish that: + +- In the class instance header, we reserve a bit `INOUT`. +- When a unique reference to a COW buffer `b` is copied into an + `inout` lvalue, we save the value of the `b.INOUT` bit and set it. +- When a reference to `b` is taken that is not part of an `inout` + value, `b.INOUT` is cleared. +- When `b` is written-back into `r`, `b.INOUT` is restored to the + saved value. +- A COW buffer can be modified in-place when it is uniquely referenced + *or* when `INOUT` is set. + +We believe this can be done with little user-facing change; the author +of a COW type would add an attribute to the property that stores the +buffer, and we would use a slightly different check for in-place +writability. + +### Other Considered Solutions + +Move optimization seemed like a potential solution when we first +considered this problem--given that it is already unspecified to +reference a property while an active `inout` reference can modify it, it +seems natural to move ownership of the value to the `inout` when +entering writeback and move it back to the original value when exiting +writeback. We do not think it is viable for the following reasons: + +- In general, relying on optimizations to provide performance + semantics is brittle. +- Move optimization would not be memory safe if either the original + value or `inout` slice were modified to give up ownership of the + original backing store. Although observing a value while it has + inout aliases is unspecified, it should remain memory-safe to do so. + This should remain memory safe, albeit unspecified: + + var arr = [1,2,3] + func mutate(x: inout Int[]) -> Int[] { + x = [3...4] + return arr[0...2] + } + mutate(&arr[0...2]) + + Inout slices thus require strong ownership of the backing store + independent of the original object, which must also keep strong + ownership of the backing store. +- Move optimization requires unique referencing and would fail when + there are multiple concurrent, non-overlapping `inout` slices. + `swap(&x.a, &x.b)` should be well-defined if `x.a` and `x.b` do not + access overlapping state, and so should be + `swap(&x[0...50], &x[50...100])`. More generally, we would like to + use inout slicing to implement divide-and-conquer parallel + algorithms, as in: + + async { mutate(&arr[0...50]) } + async { mutate(&arr[50...100]) } + +Language Changes +---------------- + +### Builtin.isUniquelyReferenced + +A mechanism needs to be exposed to library writers to allow them to +check whether a buffer is uniquely referenced. This check requires +primitive access to the layout of the heap object, and can also +potentially be reasoned about by optimizations, so it makes sense to +expose it as a `Builtin` which lowers to a SIL `is_uniquely_referenced` +instruction. + +### The `@cow` attribute + +A type may declare a stored property as being `@cow`: + + class ArrayBuffer { /* ... */ } + + struct Array { + @cow var buffer : ArrayBuffer + } + +The property must meet the following criteria: + +- It must be a stored property. +- It must be of a pure Swift class type. (More specifically, at the + implementation level, it must have a Swift refcount.) +- It must be mutable. A `@cow val` property would not be useful. + +Values with `@cow` properties have special implicit behavior when they +are used in `inout` contexts, described below. + +Implementation of @cow properties +--------------------------------- + +### inout SIL operations + +To maintain the `INOUT` bit of a class instance, we need new SIL +operations that update the `INOUT` bit. Because the state of the bit +needs to be saved and restored through every writeback scope, we can +have: + + %former = inout_retain %b : $ClassType + +increase the retain count, save the current value of `INOUT`, set +`INOUT`, and produce the `%former` value as its `Int1` result. To +release, we have: + + inout_release %b : $ClassType, %former : $Builtin.Int1 + +both reduce the retain count and change the value of `INOUT` back to the +value saved in `%former`. Furthermore: + + strong_retain %b : $ClassType + +must always clear the `INOUT` bit. + +To work with opaque types, `copy_addr` must also be able to perform an +`inout` initialization of a writeback buffer as well as reassignment to +an original value. This can be an additional attribute on the source, +mutually exclusive with `[take]`: + + copy_addr [inout] %a to [initialization] %b + +This implies that value witness tables will need witnesses for +inout-initialization and inout-reassignment. + +### Copying of @cow properties for writeback + +When a value is copied into a writeback buffer, its `@cow` properties +must be retained for the new value using `inout_retain` instead of +`strong_retain` (or `copy_addr [inout] [initialization]` instead of +plain `copy_addr [initialization]`). When the value is written back, the +property values should be `inout_release`d, or the value should be +written back using `copy_addr [inout]` reassignment. diff --git a/docs/proposals/InoutCOWOptimization.rst b/docs/proposals/InoutCOWOptimization.rst deleted file mode 100644 index cc88b59e77ffc..0000000000000 --- a/docs/proposals/InoutCOWOptimization.rst +++ /dev/null @@ -1,216 +0,0 @@ -:orphan: - -================================================ - Copy-On-Write Optimization of ``inout`` Values -================================================ - -:Authors: Dave Abrahams, Joe Groff - -:Summary: Our writeback model interacts with Copy-On-Write (COW) to - cause some surprising inefficiencies, such as O(N) performance - for ``x[0][0] = 1``. We propose a modified COW optimization - that recovers O(1) performance for these cases and supports - the efficient use of slices in algorithm implementation. - -Whence the Problem? -=================== - -The problem is caused as follows: - -* COW depends on the programmer being able to mediate all writes (so - she can copy if necessary) - -* Writes to container elements and slices are mediated through - subscript setters, so in :: - - x[0].mutate() - - we “``subscript get``” ``x[0]`` into a temporary, mutate the - temporary, and “``subscript set``” it back into ``x[0]``. - -* When the element itself is a COW type, that temporary implies a - retain count of at least 2 on the element's buffer. - -* Therefore, mutating such an element causes an expensive copy, *even - when the element's buffer isn't otherwise shared*. - -Naturally, this problem generalizes to any COW value backed by a -getter/setter pair, such as a computed or resilient ``String`` -property:: - - anObject.title.append('.') // O(N) - -Interaction With Slices -======================= - -Consider the classic divide-and-conquer algorithm QuickSort, which -could be written as follows: - -.. parsed-literal:: - - protocol Sliceable { - ... - @mutating - func quickSort(compare: (StreamType.Element, StreamType.Element)->Bool) { - let (start,end) = (startIndex, endIndex) - if start != end && start.succ() != end { - let pivot = self[start] - let mid = partition({compare($0, pivot)}) - **self[start...mid].quickSort(compare)** - **self[mid...end].quickSort(compare)** - } - } - } - -The implicit ``inout`` on the target of the recursive ``quickSort`` -calls currently forces two allocations and O(N) copies in each layer -of the QuickSort implementation. Note that this problem applies to -simple containers such as ``Int[]``, not just containers of COW -elements. - -Without solving this problem, mutating algorithms must operate on -``MutableCollection``\ s and pairs of their ``Index`` types, and we -must hope the ARC optimizer is able to eliminate the additional -reference at the top-level call. However, that does nothing for the -cases mentioned in the previous section. - -Our Solution -============ - -We need to prevent lvalues created in an ``inout`` context from -forcing a copy-on-write. To accomplish that: - -* In the class instance header, we reserve a bit ``INOUT``. - -* When a unique reference to a COW buffer ``b`` is copied into - an ``inout`` lvalue, we save the value of the ``b.INOUT`` bit and set it. - -* When a reference to ``b`` is taken that is not part of an ``inout`` - value, ``b.INOUT`` is cleared. - -* When ``b`` is written-back into ``r``, ``b.INOUT`` is restored to the saved - value. - -* A COW buffer can be modified in-place when it is uniquely referenced - *or* when ``INOUT`` is set. - -We believe this can be done with little user-facing change; the author -of a COW type would add an attribute to the property that stores the -buffer, and we would use a slightly different check for in-place -writability. - -Other Considered Solutions --------------------------- - -Move optimization seemed like a potential solution when we first considered -this problem--given that it is already unspecified to reference a property -while an active ``inout`` reference can modify it, it seems natural to move -ownership of the value to the ``inout`` when entering writeback and move it -back to the original value when exiting writeback. We do not think it is viable -for the following reasons: - -- In general, relying on optimizations to provide performance semantics is - brittle. -- Move optimization would not be memory safe if either the original value or - ``inout`` slice were modified to give up ownership of the original backing - store. Although observing a value while it has inout aliases is unspecified, - it should remain memory-safe to do so. This should remain memory safe, albeit - unspecified:: - - var arr = [1,2,3] - func mutate(x: inout Int[]) -> Int[] { - x = [3...4] - return arr[0...2] - } - mutate(&arr[0...2]) - - Inout slices thus require strong ownership of the backing store independent - of the original object, which must also keep strong ownership of the backing - store. -- Move optimization requires unique referencing and would fail when there are - multiple concurrent, non-overlapping ``inout`` slices. ``swap(&x.a, &x.b)`` - should be well-defined if ``x.a`` and ``x.b`` do not access overlapping - state, and so should be ``swap(&x[0...50], &x[50...100])``. More - generally, we would like to use inout slicing to implement - divide-and-conquer parallel algorithms, as in:: - - async { mutate(&arr[0...50]) } - async { mutate(&arr[50...100]) } - -Language Changes -================ - -Builtin.isUniquelyReferenced ----------------------------- - -A mechanism needs to be exposed to library writers to allow them to check -whether a buffer is uniquely referenced. This check requires primitive access -to the layout of the heap object, and can also potentially be reasoned about -by optimizations, so it makes sense to expose it as a ``Builtin`` which lowers -to a SIL ``is_uniquely_referenced`` instruction. - -The ``@cow`` attribute ----------------------- - -A type may declare a stored property as being ``@cow``:: - - class ArrayBuffer { /* ... */ } - - struct Array { - @cow var buffer : ArrayBuffer - } - -The property must meet the following criteria: - -- It must be a stored property. -- It must be of a pure Swift class type. (More specifically, at the - implementation level, it must have a Swift refcount.) -- It must be mutable. A ``@cow val`` property would not be useful. - -Values with ``@cow`` properties have special implicit behavior when they are -used in ``inout`` contexts, described below. - -Implementation of @cow properties -================================= - -inout SIL operations --------------------- - -To maintain the ``INOUT`` bit of a class instance, we need new SIL operations -that update the ``INOUT`` bit. Because the state of the bit needs to be -saved and restored through every writeback scope, we can have:: - - %former = inout_retain %b : $ClassType - -increase the retain count, save the current value of ``INOUT``, set ``INOUT``, -and produce the ``%former`` value as its ``Int1`` result. To release, -we have:: - - inout_release %b : $ClassType, %former : $Builtin.Int1 - -both reduce the retain count and change the value of ``INOUT`` back to the -value saved in ``%former``. Furthermore:: - - strong_retain %b : $ClassType - -must always clear the ``INOUT`` bit. - -To work with opaque types, ``copy_addr`` must also be able to perform an -``inout`` initialization of a writeback buffer as well as reassignment to -an original value. This can be an additional attribute on the source, mutually -exclusive with ``[take]``:: - - copy_addr [inout] %a to [initialization] %b - -This implies that value witness tables will need witnesses for -inout-initialization and inout-reassignment. - -Copying of @cow properties for writeback ----------------------------------------- - -When a value is copied into a writeback buffer, its ``@cow`` properties must -be retained for the new value using ``inout_retain`` instead of -``strong_retain`` (or ``copy_addr [inout] [initialization]`` instead of plain -``copy_addr [initialization]``). When the value is written back, the -property values should be ``inout_release``\ d, or the value should be -written back using ``copy_addr [inout]`` reassignment. diff --git a/docs/proposals/Inplace.md b/docs/proposals/Inplace.md new file mode 100644 index 0000000000000..ce5b84bbeb85f --- /dev/null +++ b/docs/proposals/Inplace.md @@ -0,0 +1,338 @@ +In-Place Operations +=================== + +Author + +: Dave Abrahams + +Author + +: Joe Groff + +Abstract + +: The goal of efficiently processing complex data structures leads + naturally to pairs of related operations such as `+` and `+=`: one + that produces a new value, and another that mutates on the data + structure in-place. By formalizing the relationship and adding + syntactic affordances, we can make these pairs easier to work with + and accelerate the evaluation of some common expressions. + +Examples +-------- + +In recent standard library design meetings about the proper API for +sets, it was decided that the canonical `Set` interface should be +written in terms of methods: [^1] : + + struct Set { + public func contains(x: T) -> Bool // x ∈ A, A ∋ x + public func isSubsetOf(b: Set) -> Bool // A ⊆ B + public func isStrictSubsetOf(b: Set) -> Bool // A ⊂ B + public func isSupersetOf(b: Set) -> Bool // A ⊇ B + public func isStrictSupersetOf(b: Set) -> Bool // A ⊃ B + ... + } + +When we started to look at the specifics, however, we ran into a +familiar pattern: + + ... + public func union(b: Set) -> Set // A ∪ B + public mutating func unionInPlace(b: Set) // A ∪= B + + public func intersect(b: Set) -> Set // A ∩ B + public mutating func intersectInPlace(b: Set) // A ∩= B + + public func subtract(b: Set) -> Set // A - B + public mutating func subtractInPlace(b: Set) // A -= B + + public func exclusiveOr(b: Set) -> Set // A ⊕ B + public mutating func exclusiveOrInPlace(b: Set) // A ⊕= B + +We had seen the same pattern when considering the API for `String`, but +in that case, there are no obvious operator spellings in all of Unicode. +For example: + + struct String { + public func uppercase() -> String + public mutating func uppercaseInPlace() + + public func lowercase() -> String + public mutating func lowercaseInPlace() + + public func replace( + pattern: String, with replacement: String) -> String + public mutating func replaceInPlace( + pattern: String, with replacement: String) + + public func trim() -> String + public mutating func trimInPlace() + ... + } + +It also comes up with generic algorithms such as `sort()` (which is +mutating) and `sorted()`, the corresponding non-mutating version. + +We see at least four problems with this kind of API: + +1. The lack of a uniform naming convention is problematic. People have + already complained about the asymmetry between mutating `sort()`, + and non-mutating `reverse()`. The pattern used by `sort()` and + `sorted()` doesn't apply everywhere, and penalizes the non-mutating + form, which should be the more economical of the two. +2. Naming conventions that work everywhere and penalize the mutating + form are awkward. In the case of `String` it was considered bad + enough that we didn't bother with the mutating versions of any + operations other than concatenation (which we spelled using `+` and + `+=`). +3. Producing a complete interface that defines both variants of each + operation is needlessly tedious. A working (if non-optimal) mutating + version of `op(x: T, y: U) -> T` can always be defined as : + + func opInPlace(inout x: T, y: U) { + x = op(x, y) + } + + Default implementations in protocols could do a lot to relieve + tedium here, but cranking out the same `xxxInPlace` pattern for each + `xxx` still amounts to a lot of boilerplate. + +4. Without formalizing the relationship between the mutating and + non-mutating functions, we lose optimization opportunities. For + example, it should be possible for the compiler to rewrite : + + let x = a.intersect(b).intersect(c).intersect(d) + + as : + + var t = a.intersect(b) + t.intersectInPlace(c) + t.intersectInPlace(d) + let x = t + + for efficiency, without forcing the user to sacrifice expressivity. + This optimization would generalize naturally to more common idioms + such as: + + let newString = s1 + s2 + s3 + s4 + + Given all the right conditions, it is true that a similar + optimization can be made at runtime for COW data structures using a + uniqueness check on the left-hand operand. However, that approach + only applies to COW data structures, and penalizes other cases. + +The Proposal +------------ + +Our proposal has four basic components: + +1. Solve the naming convention problem by giving the mutating and + non-mutating functions the same name. +2. Establish clarity at the point of use by extending the language to + support a concise yet distinctive syntax for invoking the + mutating operation. +3. Eliminate tedium by allowing mutating functions to be automatically + generated from non-mutating ones, and, for value types, vice-versa + (doing this for reference types is problematic due to the lack of a + standard syntax for copying the referent). +4. Support optimization by placing semantic requirements on mutating + and non-mutating versions of the same operation, and allowing the + compiler to make substitutions. + +### Use One Simple Name + +There should be one simple name for both in-place and non-mutating +sorting: `sort`. Set union should be spelled `union`. This unification +bypasses the knotty problem of naming conventions and makes code cleaner +and more readable. + +When these paired operations are free functions, we can easily +distinguish the mutating versions by the presence of the address-of +operator on the left-hand side: + + let z = union(x, y) // non-mutating + union(&x, y) // mutating + +Methods are a more interesting case, since on mutating methods, `self` +is *implicitly* `inout`: + + x.union(y) // mutating or non-mutating? + +We propose to allow method pairs of the form: + +The second `=f` method is known as an **assignment method** [^2]. +Assignment methods are implicitly `mutating`. Together these two +methods, `f` and `=f`, are known as an **assignment method pair**. This +concept generalizes in obvious ways to pairs of generic methods, details +open for discussion. + +An assignment method is only accessible via a special syntax, for +example: + +The target of an assignment method is always required, even when the +target is `self`: + + extension Set { + mutating func frob(other: Set) { + let brick = union(other) // self.union(other) implied + self.=union(other) // calls the assignment method + union(other) // warning: result ignored + } + } + +### Assignment Operator Pairs + +Many operators have assignment forms, for instance, `+` has `+=`, `-` +has `-=`, and so on. However, not all operators do; `!=` is not the +assignment form of `!`, nor is `<=` the assignment form of `<`. +Operators with assignment forms can declare this fact in their +`operator` declaration: + +For an operator *op* which `has_assignment`, a pair of operator +definitions of the form: + +is known as an **assignment operator pair**, and similar generalization +to pairs of generic operators is possible. + +To avoid confusion, the existing `assignment` operator modifier, which +indicates that an operator receives one of its operands implicitly +`inout`, shall be renamed `mutating`, since it can also be applied to +non-assignment operators: + +If an operator `op` which `has_assignment` is in scope, it is an error +to declare `op=` as an independent operator: + +Eliminating Tedious Boilerplate +------------------------------- + +### Generating the In-Place Form + +Given an ordinary method of a type `X`: + +if there is no corresponding *assignment method* in `X` with the +signature + +we can compile the statement + +as though it were written: + +### Generating the Non-Mutating Form + +Given an *assignment method* of a value type `X`: + +if there is no method in `X` with the signature + +we can compile the expression + +as though it were written: + +### Generating Operator Forms + +If only one member of an *assignment operator pair* is defined, similar +rules allow the generation of code using the other member. E.g. + +we can compile + +as though it were written: + +or + +as though it were written: + +Class Types +----------- + +Assignment and operators are generally applied to value types, but it's +reasonable to ask how to apply them to class types. The first and most +obvious requirement, in our opinion, is that immutable class types, +which are fundamentally values, should work properly. + +An assignment operator for an immutable class `X` always has the form: + +or, with COW optimization: + +Notice that compiling either form depends on an assignment to `lhs`. + +A method of a class, however, cannot assign to `self`, so no +explicitly-written assignment method can work properly for an immutable +class. Therefore, at *least* until there is a way to reseat `self` in a +method, explicitly-written assignment methods must be banned for class +types: + + // Invalid code: + class Foo { + let x: Int + required init(x: Int) { self.x = x } + + func advanced(amount: Int) -> Self { + return Self(x: self.x + amount) + } + + // Error, because we can't reseat self in a class method + func =advanced(amount: Int) { + self = Self(x: self.x + amount) + // This would also be inappropriate, since it would violate value + // semantics: + // self.x += amount + } + } + +That said, given an explicitly-written non-assignment method that +produces a new instance, the rules given above for implicitly-generated +assignment method semantics work just fine: + + // Valid code: + class Foo { + let x: Int + required init(x: Int) { self.x = x } + + func advanced(amount: Int) -> Self { + return Self(x: self.x + amount) + } + } + + var foo = Foo(x: 5) + // Still OK; exactly the same as foo = foo.advanced(10) + foo.=advanced(10) + +The alternative would be to say that explicitly-written assignment +methods cannot work properly for immutable classes and “work” with +reference semantics on other classes. We consider this approach +indefensible, especially when one considers that operators encourage +writing algorithms that can only work properly with value semantics and +will show up in protocols. + +Assignment Methods and Operators In Protocols +--------------------------------------------- + +The presence of a `=method` signature in the protocol implies that the +corresponding non-assignment signature is available. Declaring `=method` +in a protocol generates two witness table slots, one for each method of +the implied pair. If the `=method` signature is provided in the +protocol, any corresponding non-assignment `method` signature is +ignored. A type can satisfy the protocol requirement by providing either +or both members of the pair; a thunk for the missing member of the pair +is generated as needed. + +When only the non-assignment `method` member of a pair appears in the +protocol, it generates only one witness table slot. The assignment +signature is implicitly available on existentials and archetypes, with +the usual implicitly-generated semantics. + +------------------------------------------------------------------------ + +[^1]: Unicode operators, which dispatch to those methods, would also be + supported. For example, : + + public func ⊃ (a: Set, b: Set) -> Bool { + return a.isStrictSupersetOf(b) + } + + however we decided that these operators were sufficiently esoteric, + and also inaccessible using current programming tools, that they had + to remain a secondary interface. + +[^2]: the similarity to getter/setter pairs is by no means lost on the + authors. However, omitting one form in this case has a very + different meaning than in the case of getter/setter pairs. diff --git a/docs/proposals/Inplace.rst b/docs/proposals/Inplace.rst deleted file mode 100644 index 95d31e1623fc4..0000000000000 --- a/docs/proposals/Inplace.rst +++ /dev/null @@ -1,467 +0,0 @@ -:orphan: - -.. @raise litre.TestsAreMissing - -===================== - In-Place Operations -===================== - -:Author: Dave Abrahams -:Author: Joe Groff - -:Abstract: The goal of efficiently processing complex data structures - leads naturally to pairs of related operations such as ``+`` and - ``+=``: one that produces a new value, and another that mutates on - the data structure in-place. By formalizing the relationship and - adding syntactic affordances, we can make these pairs easier to work - with and accelerate the evaluation of some common expressions. - -Examples -======== - -In recent standard library design meetings about the proper API for -sets, it was decided that the canonical ``Set`` interface should be -written in terms of methods: [#operators]_ :: - - struct Set { - public func contains(x: T) -> Bool // x ∈ A, A ∋ x - public func isSubsetOf(b: Set) -> Bool // A ⊆ B - public func isStrictSubsetOf(b: Set) -> Bool // A ⊂ B - public func isSupersetOf(b: Set) -> Bool // A ⊇ B - public func isStrictSupersetOf(b: Set) -> Bool // A ⊃ B - ... - } - -When we started to look at the specifics, however, we ran into a -familiar pattern:: - - ... - public func union(b: Set) -> Set // A ∪ B - public mutating func unionInPlace(b: Set) // A ∪= B - - public func intersect(b: Set) -> Set // A ∩ B - public mutating func intersectInPlace(b: Set) // A ∩= B - - public func subtract(b: Set) -> Set // A - B - public mutating func subtractInPlace(b: Set) // A -= B - - public func exclusiveOr(b: Set) -> Set // A ⊕ B - public mutating func exclusiveOrInPlace(b: Set) // A ⊕= B - -We had seen the same pattern when considering the API for -``String``, but in that case, there are no obvious operator -spellings in all of Unicode. For example:: - - struct String { - public func uppercase() -> String - public mutating func uppercaseInPlace() - - public func lowercase() -> String - public mutating func lowercaseInPlace() - - public func replace( - pattern: String, with replacement: String) -> String - public mutating func replaceInPlace( - pattern: String, with replacement: String) - - public func trim() -> String - public mutating func trimInPlace() - ... - } - -It also comes up with generic algorithms such as ``sort()`` (which is -mutating) and ``sorted()``, the corresponding non-mutating version. - - -We see at least four problems with this kind of API: - -1. The lack of a uniform naming convention is problematic. People - have already complained about the asymmetry between mutating - ``sort()``, and non-mutating ``reverse()``. The pattern used by - ``sort()`` and ``sorted()`` doesn't apply everywhere, and penalizes - the non-mutating form, which should be the more economical of the two. - -2. Naming conventions that work everywhere and penalize the mutating - form are awkward. In the case of ``String`` it was considered bad - enough that we didn't bother with the mutating versions of any - operations other than concatenation (which we spelled using ``+`` - and ``+=``). - -3. Producing a complete interface that defines both variants of each - operation is needlessly tedious. A working (if non-optimal) - mutating version of ``op(x: T, y: U) -> T`` can always be defined - as :: - - func opInPlace(inout x: T, y: U) { - x = op(x, y) - } - - Default implementations in protocols could do a lot to relieve - tedium here, but cranking out the same ``xxxInPlace`` pattern for - each ``xxx`` still amounts to a lot of boilerplate. - -4. Without formalizing the relationship between the mutating and - non-mutating functions, we lose optimization opportunities. For - example, it should be possible for the compiler to rewrite :: - - let x = a.intersect(b).intersect(c).intersect(d) - - as :: - - var t = a.intersect(b) - t.intersectInPlace(c) - t.intersectInPlace(d) - let x = t - - for efficiency, without forcing the user to sacrifice expressivity. - This optimization would generalize naturally to more common idioms - such as:: - - let newString = s1 + s2 + s3 + s4 - - Given all the right conditions, it is true that a similar - optimization can be made at runtime for COW data structures using a - uniqueness check on the left-hand operand. However, that approach - only applies to COW data structures, and penalizes other cases. - -The Proposal -============ - -Our proposal has four basic components: - -1. Solve the naming convention problem by giving the mutating and - non-mutating functions the same name. - -2. Establish clarity at the point of use by extending the language to - support a concise yet distinctive syntax for invoking the mutating - operation. - -3. Eliminate tedium by allowing mutating functions to be automatically - generated from non-mutating ones, and, for value types, vice-versa - (doing this for reference types is problematic due to the lack of a - standard syntax for copying the referent). - -4. Support optimization by placing semantic requirements on mutating - and non-mutating versions of the same operation, and allowing the - compiler to make substitutions. - -Use One Simple Name -------------------- - -There should be one simple name for both in-place and non-mutating -sorting: ``sort``. Set union should be spelled ``union``. This -unification bypasses the knotty problem of naming conventions and -makes code cleaner and more readable. - -When these paired operations are free functions, we can easily -distinguish the mutating versions by the presence of the address-of -operator on the left-hand side:: - - let z = union(x, y) // non-mutating - union(&x, y) // mutating - -Methods are a more interesting case, since on mutating methods, -``self`` is *implicitly* ``inout``:: - - x.union(y) // mutating or non-mutating? - -We propose to allow method pairs of the form: - -.. parsed-literal:: - - extension **X** { - func *f*\ (p₀: T₀, p₁: T₁, p₂: T₂, …p\ *n*: T\ *n*) -> **X** - - func **=**\ *f*\ (p₀: T₀, p₁: T₁, p₂: T₂, …p\ *n*: T\ *n*) -> **Void** - } - -The second ``=f`` method is known as an **assignment method** [#getset]_. -Assignment methods are implicitly ``mutating``. -Together these two methods, ``f`` and ``=f``, are known as an -**assignment method pair**. This concept generalizes in obvious ways -to pairs of generic methods, details open for discussion. - -An assignment method is only accessible via a special syntax, for -example: - -.. parsed-literal:: - - x\ **.=**\ union(y) - -The target of an assignment method is always required, even when the -target is ``self``:: - - extension Set { - mutating func frob(other: Set) { - let brick = union(other) // self.union(other) implied - self.=union(other) // calls the assignment method - union(other) // warning: result ignored - } - } - -Assignment Operator Pairs -------------------------- - -Many operators have assignment forms, for instance, ``+`` has ``+=``, ``-`` -has ``-=``, and so on. However, not all operators do; ``!=`` is not the -assignment form of ``!``, nor is ``<=`` the assignment form of ``<``. Operators -with assignment forms can declare this fact in their ``operator`` declaration: - -.. parsed-literal:: - - infix operator + { - **has_assignment** - } - -For an operator *op* which ``has_assignment``, a pair of operator definitions -of the form: - -.. parsed-literal:: - - func *op*\ (**X**, Y) -> **X** - - func *op*\ =(**inout X**, Y) -> **Void** - -is known as an **assignment operator pair**, and similar -generalization to pairs of generic operators is possible. - -To avoid confusion, the existing ``assignment`` operator modifier, which -indicates that an operator receives one of its operands implicitly ``inout``, -shall be renamed ``mutating``, since it can also be applied to non-assignment -operators: - -.. parsed-literal:: - - postfix operator ++ { - **mutating** // formerly "assignment" - } - -If an operator ``op`` which ``has_assignment`` is in scope, it is an error to -declare ``op=`` as an independent operator: - -.. parsed-literal:: - - operator *☃* { has_assignment } - - // Error: '☃=' is the assignment form of existing operator '☃' - operator *☃=* { has_assignment } - -Eliminating Tedious Boilerplate -=============================== - -Generating the In-Place Form ----------------------------- - -Given an ordinary method of a type ``X``: - -.. parsed-literal:: - - extension **X** { - func *f*\ (p₀: T₀, p₁: T₁, p₂: T₂, …p\ *n*: T\ *n*) -> **X** - } - -if there is no corresponding *assignment method* in ``X`` with the signature - -.. parsed-literal:: - - extension **X** { - func *=f*\ (p₀: T₀, p₁: T₁, p₂: T₂, …p\ *n*: T\ *n*) -> **Void** - } - -we can compile the statement - -.. parsed-literal:: - - x\ **.=**\ *f*\ (a₀, p₁: a₁, p₂: a₂, …p\ *n*: a\ *n*) - -as though it were written: - -.. parsed-literal:: - - x **= x.**\ *f*\ (a₀, p₁: a₁, p₂: a₂, …p\ *n*: a\ *n*) - -Generating the Non-Mutating Form --------------------------------- - -Given an *assignment method* of a value type ``X``: - -.. parsed-literal:: - - extension **X** { - func *=f*\ (p₀: T₀, p₁: T₁, p₂: T₂, …p\ *n*: T\ *n*) -> **Void** - } - -if there is no method in ``X`` with the signature - -.. parsed-literal:: - - extension **X** { - func *f*\ (p₀: T₀, p₁: T₁, p₂: T₂, …p\ *n*: T\ *n*) -> **X** - } - -we can compile the expression - -.. parsed-literal:: - - **x.**\ *f*\ (a₀, p₁: a₁, p₂: a₂, …p\ *n*: a\ *n*) - -as though it were written: - -.. parsed-literal:: - - { - (var y: X)->X in - y\ **.=**\ *f*\ (a₀, p₁: a₁, p₂: a₂, …p\ *n*: a\ *n*) - return y - }(x) - -Generating Operator Forms -------------------------- - -If only one member of an *assignment operator pair* is defined, similar -rules allow the generation of code using the other member. E.g. - -we can compile - -.. parsed-literal:: - - x *op*\ **=** *expression* - -as though it were written: - -.. parsed-literal:: - - x **=** x *op* (*expression*) - -or - -.. parsed-literal:: - - x *op* *expression* - -as though it were written: - -.. parsed-literal:: - - { - (var y: X)->X in - y *op*\ **=**\ *expression* - return y - }(x) - -Class Types -=========== - -Assignment and operators are generally applied to value types, but -it's reasonable to ask how to apply them to class types. The first -and most obvious requirement, in our opinion, is that immutable class -types, which are fundamentally values, should work properly. - -An assignment operator for an immutable class ``X`` always has the form: - -.. parsed-literal:: - - func *op*\ **=** (**inout** lhs: X, rhs: Y) { - lhs = *expression creating a new X object* - } - -or, with COW optimization: - -.. parsed-literal:: - - func *op*\ **=** (**inout** lhs: X, rhs: Y) { - if isUniquelyReferenced(&lhs) { - lhs.\ *mutateInPlace*\ (rhs) - } - else { - lhs = *expression creating a new X object* - } - } - -Notice that compiling either form depends on an assignment to ``lhs``. - -A method of a class, however, cannot assign to ``self``, so no -explicitly-written assignment method can work properly for an -immutable class. Therefore, at *least* until there is a way to reseat ``self`` -in a method, explicitly-written assignment methods must be banned for -class types:: - - // Invalid code: - class Foo { - let x: Int - required init(x: Int) { self.x = x } - - func advanced(amount: Int) -> Self { - return Self(x: self.x + amount) - } - - // Error, because we can't reseat self in a class method - func =advanced(amount: Int) { - self = Self(x: self.x + amount) - // This would also be inappropriate, since it would violate value - // semantics: - // self.x += amount - } - } - -That said, given an explicitly-written -non-assignment method that produces a new instance, the rules given -above for implicitly-generated assignment method semantics work just -fine:: - - // Valid code: - class Foo { - let x: Int - required init(x: Int) { self.x = x } - - func advanced(amount: Int) -> Self { - return Self(x: self.x + amount) - } - } - - var foo = Foo(x: 5) - // Still OK; exactly the same as foo = foo.advanced(10) - foo.=advanced(10) - -The alternative would be to say that explicitly-written assignment methods -cannot work properly for immutable classes and “work” with reference -semantics on other classes. We consider this approach indefensible, -especially when one considers that operators encourage writing -algorithms that can only work properly with value semantics and will -show up in protocols. - -Assignment Methods and Operators In Protocols -============================================= - -The presence of a ``=method`` signature in the protocol implies that -the corresponding non-assignment signature is available. Declaring -``=method`` in a protocol generates two witness table -slots, one for each method of the implied pair. If the -``=method`` signature is provided in the protocol, any -corresponding non-assignment ``method`` signature is ignored. A type can -satisfy the protocol requirement by providing either or both members -of the pair; a thunk for the missing member of the pair is generated -as needed. - -When only the non-assignment ``method`` member of a pair appears in the -protocol, it generates only one witness table slot. The assignment -signature is implicitly available on existentials and archetypes, with -the usual implicitly-generated semantics. - ----------- - -.. [#operators] Unicode operators, which dispatch to those methods, - would also be supported. For example, :: - - public func ⊃ (a: Set, b: Set) -> Bool { - return a.isStrictSupersetOf(b) - } - - however we decided that these operators were sufficiently esoteric, - and also inaccessible using current programming tools, that they - had to remain a secondary interface. - -.. [#getset] the similarity to getter/setter pairs is by no means lost on - the authors. However, omitting one form in this case has a - very different meaning than in the case of getter/setter - pairs. diff --git a/docs/proposals/ObjC Interoperation.md b/docs/proposals/ObjC Interoperation.md new file mode 100644 index 0000000000000..262c9fbd3887b --- /dev/null +++ b/docs/proposals/ObjC Interoperation.md @@ -0,0 +1,480 @@ +Interaction with Objective-C +============================ + +Authors + +: John McCall + +I propose some elementary semantics and limitations when exposing Swift +code to Objective C and vice-versa. + +Dynamism in Objective-C +----------------------- + +Objective C intentionally defines its semantics almost entirely around +its implementation model. The implementation supports two basic +operations that you can perform on any object: + +### Instance variable access + +An access to an ivar `foo` is type-checked by looking for a declared +ivar with the name `foo` in the static type of the base object. To +succeed, that search must find an ivar from some class `C`. + +It is undefined behavior if, at runtime, the base expression does not +evaluate to a valid object of type `C` or some subclass thereof. This +restriction is necessary to allow the compiler to compute the address of +the ivar at (relatively) minimal expense. + +Because of that restriction, ivar accesses are generally seen by users +as "primitive", and we probably have some additional flexibility about +optimizing them. For example, we could probably impose a rule insisting +that the entire static type of the base be accurate, thus forbidding a +user from accessing an ivar using a pointer to an incorrect subclass of +`C`. This would theoretically allow stronger alias analysis: for +example, if we have `D *d;` and `E* e;` where those types are subclasses +of `C`, we would be able to prove that `d->foo` does not alias `e->foo`. +However, in practice, ObjC ivars tend to be directly accessed only +within the implementation of the class which defines them, which means +that the stronger rule would basically never kick in. + +### Message sends + +A message send is type-checked by looking for a declared method with +that selector in the static type of the receiver, or, if the receiver +has type `id` or `Class`, in any known class in the translation unit. +The arguments and expression result are then determined by that method +declaration. + +At runtime, if the receiver is actually `nil`, the message send returns +immediately with a zero-initialized result; otherwise the runtime +searches for a method implementation (and, sometimes, an alternate +receiver) via a somewhat elaborate algorithm: + +> - The runtime first searches the method tables of the object for a +> method with that selector, starting from the object's dynamic +> class and proceeding up the hierarchy. +> - If that search fails, the runtime gives the class an opportunity +> to dynamically resolve the method by adding it to the method lists +> of one of the classes in the hierarchy. +> - If dynamic resolution fails, the runtime invokes the forwarding +> handler; the standard handler installed by CoreFoundation follows +> a complex rule that needn't be described here other than to note +> that it can involve actually sending the message to a completely +> different object if the receiver agrees. + +It is undefined behavior if the signature of the method implementation +actually found is not compatible with the signature of the method +against which the message send was type-checked. This restriction is +necessary in order to avoid the need to perform dynamic signature +checking and automatic implicit conversion on arguments, which would +substantially slow down the method-call mechanism. + +Note that common practice requires this sense of "compatible" to be much +looser than C's. For example, `performSelector` is documented as +expecting the target method to return an object reference, but it is +regularly used to invoke methods that actually return `void`. + +Otherwise, the language provides no guarantees which would allow the +compiler to reason accurately about what the invoked method will +actually do. Moreover, this is not just a formal oversight; there is +quite a bit of code relying on the ability to break each of these +non-guarantees. We can classify them into five categories: + +> - The language does not guarantee that the static type of an object +> reference will be accurate. Just because a value is typed `C*` +> does not mean it is dynamically a reference to a `C` object. Proxy +> objects are frequently passed around as if they were values of +> their target class. (Some people would argue that users should +> only proxy protocols, not entire concrete classes; but that's not +> universally followed.) +> - The language does not guarantee that the complete class hierarchy +> is statically knowable. Even ignoring the limitations of the C +> compilation model, it is possible to dynamically construct classes +> with new methods that may override the declared behavior of +> the class. Core Data relies on this kind of dynamic +> class generation. +> - The language does not guarantee that an object's dynamic class +> will be the type that a user actually appeared to allocate. Even +> ignoring proxies, it is relatively common for a factory method on +> a class to return an instance of a subclass. +> - The language does not guarantee that an object's dynamic class +> will not change. KVO works by changing an object's dynamic class +> to a dynamically-generated subclass with new methods that replace +> the observed accessors. +> +> However, it is probably reasonable to call it undefined behavior +> to change a dynamic class in a way that would add or remove ivars. +> +> - The language does not guarantee that a class will remain constant. +> Loading a new dynamic library may introduce categories that add +> and replace methods on existing classes, and the runtime provides +> public functions to do the same. These features are often used to +> introduce dynamic instrumentation, for example when debugging. + +All of these constraints combined --- actually, many of them +individually --- make devirtualization completely impossible in +Objective-C [^1]. + +Devirtualization in Swift +------------------------- + +Method devirtualization [^2] is likely to be a critically important +optimization in Swift. + +### A Missing Optimization + +For one, it is an important missing optimization even in Objective C. +Any program that tries to separate its concerns will usually introduce +some extra abstraction in its formal model. For example: + +> - A class might provide multiple convenience initializers that all +> delegate to each other so that all initialization will flow +> through a single point. +> - A large operation might be simpler to reason about when split into +> several smaller methods. +> - A property might be abstracted behind a getter/setter to make it +> easier to change the representation (or do additional work on set) +> later. + +In each of the examples, the user has made a totally reasonable decision +about code organization and reserved flexibility, and Objective C +proceeds to introduce unnecessary runtime costs which might force a +performance-sensitive programmer to choose a different path. + +### Swift-Specific Concerns + +The lack of devirtualization would hit Swift much harder because of its +property model. With a synthesized property, Objective C provides a way +to either call the getter/setter (with dot syntax) or directly access +the underlying ivar (with arrow syntax). By design, Swift hides that +difference, and the abstract language model is that all accesses go +through a getter or setter. + +Using a getter or setter instead of a direct access is a major +regression for several reasons. The first is the direct one: the +generated code must call a function, which prevents the compiler from +keep values live in the most efficient way, and which inhibits most +compiler analyses. The second is a by-product of value types: if a value +is read, modified, and then written back, the modification will take +place on the temporary copy, forcing a copy-on-write. Any proposal to +improve on that relies on having a richer API for the access than merely +a getter/setter pair, which cannot be guaranteed. + +For properties of a value type, this isn't a performance problem, +because we can simply look at the implementation (ignoring resilience +for now) and determine whether we can access the property directly. But +for properties of a class type, polymorphism requires us to defensively +handle the possibility that a subclass might add arbitrary logic to +either the getter or setter. If our implementation model is as +unrestricted as Objective C's, that's a serious problem. + +I think that this is such a massive regression from Objective C that we +have to address it. + +### Requirements for Devirtualization + +There are several different ways to achieve devirtualization, each with +its own specific requirements. But they all rely on a common guarantee: +we must remove or constrain the ability to dynamically add and replace +method implementations. + +#### Restricting Method Replacement + +There are two supported ways to add or replace methods in Objective C. + +The first is via the runtime API. If we do have to support doing this to +replace Swift methods --- and we should try to avoid that --- then I +think restricting it to require a `@dynamic` annotation on the +replaceable method (or its lexical context) is reasonable. We should try +to get the Objective C runtime to complain about attempts to replace +non-dynamic methods. + +The second is via categories. It's generally understood that a category +replacing an existing method implementation is "rude" + +#### Point of Allocation + +If we can see the true point of allocation of an object, then we know +its dynamic class at that point. However: + +Note that the true point of allocation is not the point at which we call +some factory method on a specific type; it has to be an actual +allocation: an `alloc_ref` SIL instruction. And it's questionable +whether an ObjC allocation counts, because those sometimes don't return +what you might expect. + +Once you know the dynamic class at a particular point, you can +devirtualize calls if: + +> - There is no supported way to replace a function implementation. + +> `+[NSManagedObject alloc]` does this). + +We can reason forward from the point of allocation. + +> If we can see that an object was allocated with `alloc_object`, then +> we know the dynamic class at that point. That's relatively easy to +> deal with. +> +> - + +> If we can restrict the ability to +> +> : change the dynamic class, or at least restrict +> +### Access Control + +Swift does give us one big tool for devirtualization that Objective C +lacks: access control. In Swift, access control determines visibility, +and it doesn't make sense to override something that you can't see. +Therefore: + +> - A declaration which is private to a file can only be overridden +> within that file. +> - A declaration which is private to a module can only be overridden +> with that module. +> - A public declaration can be overridden anywhere [^3]. + +This means that a private stored property can always be "devirtualized" +into a direct access [^4]. Unfortunately, `private` is not the default +access control: module-private is. And if the current module can contain +Objective C code, then even that raises the question of what ObjC +interop actually means. + +Using Swift Classes from Objective C +------------------------------------ + +open the question of + +Because we intentionally hide the difference between a stored property +and its underlying storage, + +For another example, code class might access + +In both cases, t makes sense to organize the code that way, but +Objective C punishes the performance of that code in order to reserve +the language's + +provide some abstractions which are unnecessary in the current +implementation. For example, a class might have multiple initializers, +each chaining to another, so that it can be conveniently constructed +with any set of arguments but any special initialization logic can go in +one place. + +Reserving that flexibility in the code is often good sense, and +reserving it across API boundaries is good language design, but it's +silly not actually + +Well-factored object-oriented code often contains a large number of +abstractions that improve the organization of the code or make it easier +to later extend or maintain, but serve no current purpose. + +In typical object-oriented code, many operations are split into several +small methods in order to improve code organization and reserve the +ability to + +Conscientious developers + +runtime calls cached-offset calculation for the ivar location. + +restriction, there's general acceptance that the + +is necessary to make ivar accesses not ridiculously expensive. Because +of that, there's general acceptance that + +If the compiler team cared, we could *probably* convince people to +accept a language rule that requires the entire static type to be +accurate, so that e.g. + +Otherwise Objective-C provides no restrictions beyond those imposed by +the source language. + +ivar is accessed on an object reference which is `nil` or otherwise not +a valid object of the class + +- You can send an object reference a message. + + - If the reference is actually `nil`, the message send returns + immediately with a zero-initialized result. + - Otherwise, the runtime searches the class hierarchy of the + object, starting from the object's dynamic class and proceeding + to superclasses, looking for a concrete method matching the + selector of the message. If a concrete method is found, it + is called. + - If the class hierarchy contains no such method, a different + method is invoked on the class to give it an opportunity to + dynamically resolve the method; if taken, the process repeats, + but skipping this step the second time. + - Otherwise, the global forwarding handler is invoked. On Darwin, + this is set up by CoreFoundation, and it follows a complicated + protocol which relies on NSInvocation. + + It is undefined behavior if the type signature of the method + implementation ultimately found is not compatible with the type + signature of the method that was used to statically type-check the + message send. This includes semantic annotations like ARC ownership + conventions and the `noreturn` attribute. Otherwise, there are no + semantic restrictions on what any particular method can do. + +> signature of the method implementation's +> +> : pr signature is not compatible with the signature at which the +> method was invoked. +> +, in which case the runtime searches + +: the class hierarchy of the object, from most to least derived, and + calls the method + +In Objective-C, every object has a class and every class has a +collection of methods. The high-level semantics are essentially those + +> We propose a new attribute, `@public`, that can adorn any declaration +> not local to a function. For the purpose of standard library +> development, even just parsing this attribute without implementing +> semantics would be extremely useful in the near term. + +Basic Semantics +--------------- + +`@public` makes a declaration visible in code where the enclosing module +is imported. So, given this declaration in the `Satchel` module: + + @public struct Bag : ... { + ... + } + +We could write, in any other module, : + + import Satchel + typealias SwingMe = Bag + +The difference from the status quo being that without `@public` on the +declaration of `Bag`, the use of `Bag` above would be ill-formed. + +Type-Checking +------------- + +The types of all parameters and the return type of a func marked +`@public` (including the implicit `self` of methods) must also be +`@public`. + +All parameters to a `func` marked `@public` (including the implicit +`self` of methods) must also be `@public`: + + struct X {} // not @public + @public struct Y {} + func f(_: X) {} // OK; also not @public + @public func g(_: Y) {} // OK; uses only @public types + @public func h(_: X, _: Y) {} // Ill-formed; non-public X in public signature + +A `typealias` marked `@public` must refer to a type marked `@public`: + + typealias XX = X // OK; not @public + @public typealias YY = Y // OK; Y is @public + @public typealias XXX = X // Ill-formed; public typealias refers to non-public type + +There is a straightforward and obvious rule for composing the +`@public`-ness of any compound type, including function types, tuple +types and instances of generic types: The compound type is public if and +only if all of the component types, are `@public` and either defined in +this module or re-exported from this module. + +Enums +----- + +The cases of an `enum` are `@public` if and only if the `enum` is +declared `@public`. + +Derived Classes +--------------- + +A method that overrides an `@public` method must be declared `@public`, +even if the enclosing class is non-`@public`. + +Protocols +--------- + +A `@public` protocol can have `@public` and non-`@public` requirements. +`@public` requirements can only be satisfied by `@public` declarations. +Non-`@public` requirements can be satisfied by `@public` or +non-`@public` declarations. + +Conformances +------------ + +The conformance of a type to a protocol is `@public` if that conformance +is part of an `@public` declaration. The program is ill-formed if any +declaration required to satisfy a `@public` conformance is not also +declared `@public`.: + + @public protocol P { + @public func f() { g() } + func g() + } + + struct X : P { // OK, X is not @public, so neither is its + func f() {} // conformance to P, and therefore f + func g() {} // can be non-@public + } + + protocol P1 {} + + @public struct Y : P1 {} // Y is @public so its + // conformance to P1 is, too. + + @public + extension Y : P { // This extension is @public, so + @public func f() {} // Y's conformance to P is also, and + func g() {} // thus f must be @public too + } + + protocol P2 {} + + extension Y : P2 {} // Y's conformance to P2 is non-@public + +> **note** +> +> It's our expectation that in the near term, and probably for + +> v1.0, non-`@public` conformances on `@public` types will be diagnosed +> as ill-formed/unsupported. + +A Related Naming Change +----------------------- + +The existing `@exported` attribute for imports should be renamed +`@public` with no change in functionality. + +Future Directions +----------------- + +Some obvious directions to go in this feature space, which we are not +proposing today, but with which we tried to make this proposal +compatible: + +- non-`@public` conformances +- file-private accessibility +- explicit non-`@public` overrides, e.g. `@!public` + +[^1]: Not completely. We could optimistically apply techniques typically + used in dynamic language implementations. For example, we could + directly call an expected method body, then guard that call with a + check of the actual dispatched implementation against the expected + method pointer. But since the inlined code would necessarily be one + side of a "diamond" in the CFG, and the branches in that diamond + would overwhelmingly be unthreadable, it is not clear that the + optimization would gain much, and it would significantly bloat the + call. + +[^2]: In contrast to generic or existential devirtualization, which are + also important, but which aren't affected by the Objective C + interoperation model. + +[^3]: We've talked about having two levels of public access control for + classes, so that you have to opt in to subclassability. I still + think this is a good idea. + +[^4]: Assuming we don't introduce a supported way of dynamically + replacing the implementation of a private Swift method! diff --git a/docs/proposals/ObjC Interoperation.rst b/docs/proposals/ObjC Interoperation.rst deleted file mode 100644 index 2eb2586c4b591..0000000000000 --- a/docs/proposals/ObjC Interoperation.rst +++ /dev/null @@ -1,563 +0,0 @@ -:orphan: - -============================== - Interaction with Objective-C -============================== - -:Authors: John McCall - -I propose some elementary semantics and limitations when exposing -Swift code to Objective C and vice-versa. - -Dynamism in Objective-C -======================= - -Objective C intentionally defines its semantics almost entirely around -its implementation model. The implementation supports two basic -operations that you can perform on any object: - -Instance variable access ------------------------- - -An access to an ivar ``foo`` is type-checked by looking for a declared -ivar with the name ``foo`` in the static type of the base object. To -succeed, that search must find an ivar from some class ``C``. - -It is undefined behavior if, at runtime, the base expression does not -evaluate to a valid object of type ``C`` or some subclass thereof. -This restriction is necessary to allow the compiler to compute the -address of the ivar at (relatively) minimal expense. - -Because of that restriction, ivar accesses are generally seen by users -as "primitive", and we probably have some additional flexibility about -optimizing them. For example, we could probably impose a rule -insisting that the entire static type of the base be accurate, thus -forbidding a user from accessing an ivar using a pointer to an -incorrect subclass of ``C``. This would theoretically allow stronger -alias analysis: for example, if we have ``D *d;`` and ``E* e;`` where -those types are subclasses of ``C``, we would be able to prove that -``d->foo`` does not alias ``e->foo``. However, in practice, ObjC -ivars tend to be directly accessed only within the implementation of -the class which defines them, which means that the stronger rule would -basically never kick in. - -Message sends -------------- - -A message send is type-checked by looking for a declared method with -that selector in the static type of the receiver, or, if the receiver -has type ``id`` or ``Class``, in any known class in the translation -unit. The arguments and expression result are then determined by that -method declaration. - -At runtime, if the receiver is actually ``nil``, the message send -returns immediately with a zero-initialized result; otherwise the -runtime searches for a method implementation (and, sometimes, an -alternate receiver) via a somewhat elaborate algorithm: - - * The runtime first searches the method tables of the object for a - method with that selector, starting from the object's dynamic - class and proceeding up the hierarchy. - - * If that search fails, the runtime gives the class an opportunity - to dynamically resolve the method by adding it to the method lists - of one of the classes in the hierarchy. - - * If dynamic resolution fails, the runtime invokes the forwarding - handler; the standard handler installed by CoreFoundation follows - a complex rule that needn't be described here other than to note - that it can involve actually sending the message to a completely - different object if the receiver agrees. - -It is undefined behavior if the signature of the method implementation -actually found is not compatible with the signature of the method -against which the message send was type-checked. This restriction is -necessary in order to avoid the need to perform dynamic signature -checking and automatic implicit conversion on arguments, which would -substantially slow down the method-call mechanism. - -Note that common practice requires this sense of "compatible" to be -much looser than C's. For example, ``performSelector`` is documented -as expecting the target method to return an object reference, but it -is regularly used to invoke methods that actually return ``void``. - -Otherwise, the language provides no guarantees which would allow the -compiler to reason accurately about what the invoked method will -actually do. Moreover, this is not just a formal oversight; there is -quite a bit of code relying on the ability to break each of these -non-guarantees. We can classify them into five categories: - - * The language does not guarantee that the static type of an object - reference will be accurate. Just because a value is typed ``C*`` - does not mean it is dynamically a reference to a ``C`` object. - Proxy objects are frequently passed around as if they were values - of their target class. (Some people would argue that users - should only proxy protocols, not entire concrete classes; but - that's not universally followed.) - - * The language does not guarantee that the complete class hierarchy - is statically knowable. Even ignoring the limitations of the C - compilation model, it is possible to dynamically construct classes - with new methods that may override the declared behavior of the - class. Core Data relies on this kind of dynamic class generation. - - * The language does not guarantee that an object's dynamic class - will be the type that a user actually appeared to allocate. Even - ignoring proxies, it is relatively common for a factory method on - a class to return an instance of a subclass. - - * The language does not guarantee that an object's dynamic class - will not change. KVO works by changing an object's dynamic class - to a dynamically-generated subclass with new methods that replace - the observed accessors. - - However, it is probably reasonable to call it undefined behavior - to change a dynamic class in a way that would add or remove ivars. - - * The language does not guarantee that a class will remain constant. - Loading a new dynamic library may introduce categories that add - and replace methods on existing classes, and the runtime provides - public functions to do the same. These features are often used to - introduce dynamic instrumentation, for example when debugging. - -All of these constraints combined --- actually, many of them -individually --- make devirtualization completely impossible in -Objective-C [1]_. - -.. [1] Not completely. We could optimistically apply techniques - typically used in dynamic language implementations. For - example, we could directly call an expected method body, then - guard that call with a check of the actual dispatched - implementation against the expected method pointer. But since - the inlined code would necessarily be one side of a "diamond" - in the CFG, and the branches in that diamond would - overwhelmingly be unthreadable, it is not clear that the - optimization would gain much, and it would significantly bloat - the call. - -Devirtualization in Swift -========================= - -Method devirtualization [2]_ is likely to be a critically important -optimization in Swift. - -.. [2] In contrast to generic or existential devirtualization, which - are also important, but which aren't affected by the Objective C - interoperation model. - -A Missing Optimization ----------------------- - -For one, it is an important missing optimization even in Objective C. -Any program that tries to separate its concerns will usually introduce -some extra abstraction in its formal model. For example: - - * A class might provide multiple convenience initializers that all - delegate to each other so that all initialization will flow - through a single point. - - * A large operation might be simpler to reason about when split into - several smaller methods. - - * A property might be abstracted behind a getter/setter to make it - easier to change the representation (or do additional work on set) - later. - -In each of the examples, the user has made a totally reasonable -decision about code organization and reserved flexibility, and -Objective C proceeds to introduce unnecessary runtime costs which -might force a performance-sensitive programmer to choose a different -path. - -Swift-Specific Concerns ------------------------ - -The lack of devirtualization would hit Swift much harder because of -its property model. With a synthesized property, Objective C provides -a way to either call the getter/setter (with dot syntax) or directly -access the underlying ivar (with arrow syntax). By design, Swift -hides that difference, and the abstract language model is that all -accesses go through a getter or setter. - -Using a getter or setter instead of a direct access is a major -regression for several reasons. The first is the direct one: the -generated code must call a function, which prevents the compiler from -keep values live in the most efficient way, and which inhibits most -compiler analyses. The second is a by-product of value types: if a -value is read, modified, and then written back, the modification will -take place on the temporary copy, forcing a copy-on-write. Any -proposal to improve on that relies on having a richer API for the -access than merely a getter/setter pair, which cannot be guaranteed. - -For properties of a value type, this isn't a performance problem, -because we can simply look at the implementation (ignoring resilience -for now) and determine whether we can access the property directly. -But for properties of a class type, polymorphism requires us to -defensively handle the possibility that a subclass might add arbitrary -logic to either the getter or setter. If our implementation model -is as unrestricted as Objective C's, that's a serious problem. - -I think that this is such a massive regression from Objective C that -we have to address it. - -Requirements for Devirtualization ---------------------------------- - -There are several different ways to achieve devirtualization, each -with its own specific requirements. But they all rely on a common -guarantee: we must remove or constrain the ability to dynamically -add and replace method implementations. - -Restricting Method Replacement -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -There are two supported ways to add or replace methods in Objective C. - -The first is via the runtime API. If we do have to support doing this -to replace Swift methods --- and we should try to avoid that --- then -I think restricting it to require a ``@dynamic`` annotation on the -replaceable method (or its lexical context) is reasonable. We should -try to get the Objective C runtime to complain about attempts to -replace non-dynamic methods. - -The second is via categories. It's generally understood that a -category replacing an existing method implementation is "rude" - - - -.. The rest of this doesn't seem to be coherent and isn't parseable as - ReST, which breaks the build - - It's arguable whether we should - even support that at all. If we do, I think that restricting it to - require some sort of ``@dynamic`` annotation on the replaceable method - (or its lexical context) is much of a problem. - - is a I don't think that restricting this is - actually a serious problem, if we . We can have some sort of - ``@dynamic`` annotation for - - I don't think that requiring some sort of ``@dynamic`` - - That one, central restriction is that we must remove or constrain the - ability to dynamically add and replace method implementations on - existing classes. It's reasonable to request some sort of ``@dynamic`` - annotation for cases where this is absolutely required. - - One interesting corner case - - don't think anybody will weep too heavily if we scale back those ObjC - runtime functions to say that either you can't use them on Swift classes - or - - restriction: removing the general ability to dynamically add and - replace method implementations on an existing class. - - There's a tension here. - -Point of Allocation -~~~~~~~~~~~~~~~~~~~ - -If we can see the true point of allocation of an object, then we know -its dynamic class at that point. However: - -Note that the true point of allocation is not the point at which we -call some factory method on a specific type; it has to be an actual -allocation: an ``alloc_ref`` SIL instruction. And it's questionable -whether an ObjC allocation counts, because those sometimes don't -return what you might expect. - -Once you know the dynamic class at a particular point, you can -devirtualize calls if: - - * There is no supported way to replace a function implementation. - - ``+[NSManagedObject alloc]`` does this). - -We can reason forward from the point of allocation. - - If we can see that an object was allocated with ``alloc_object``, - then we know the dynamic class at that point. That's relatively - easy to deal with. - - * - - If we can restrict the ability to - change the dynamic class, or at least restrict - - -Access Control --------------- - -Swift does give us one big tool for devirtualization that Objective C -lacks: access control. In Swift, access control determines -visibility, and it doesn't make sense to override something that you -can't see. Therefore: - - * A declaration which is private to a file can only be overridden - within that file. - - * A declaration which is private to a module can only be overridden - with that module. - - * A public declaration can be overridden anywhere [3]_. - -.. [3] We've talked about having two levels of public access control - for classes, so that you have to opt in to subclassability. I - still think this is a good idea. - -This means that a private stored property can always be -"devirtualized" into a direct access [4]_. Unfortunately, ``private`` -is not the default access control: module-private is. And if the -current module can contain Objective C code, then even that raises the -question of what ObjC interop actually means. - -.. [4] Assuming we don't introduce a supported way of dynamically - replacing the implementation of a private Swift method! - -Using Swift Classes from Objective C -==================================== - - - -open the question of - - -Because we intentionally hide the -difference between a stored property and its underlying storage, - - -For another example, code -class might access - -In both cases, t makes sense to organize the code that way, -but Objective C punishes the performance of that code in order to -reserve the language's - - - - -provide some abstractions which are unnecessary in the current -implementation. For example, a class might have multiple -initializers, each chaining to another, so that it can be conveniently -constructed with any set of arguments but any special initialization -logic can go in one place. - -Reserving that -flexibility in the code is often good sense, and reserving it across -API boundaries is good language design, but it's silly -not actually - -Well-factored object-oriented code often contains a large number of -abstractions that improve the organization of the code or make it -easier to later extend or maintain, but serve no current purpose. - - -In -typical object-oriented code, many operations are split into several -small methods in order to improve code organization and reserve the -ability to - -Conscientious developers - - - - - - - - - -runtime calls -cached-offset calculation for the ivar location. - -restriction, there's general acceptance that the - -is necessary to make ivar -accesses not ridiculously expensive. Because of that, there's general -acceptance that - -If the compiler team cared, we could *probably* convince people to -accept a language rule that requires the entire static type to be -accurate, so that e.g. - -Otherwise Objective-C provides no restrictions beyond those imposed by -the source language. - - -ivar is accessed on an object reference which is ``nil`` or otherwise -not a valid object of the class - - -* You can send an object reference a message. - - * If the reference is actually ``nil``, the message send returns - immediately with a zero-initialized result. - - * Otherwise, the runtime searches the class hierarchy of the object, - starting from the object's dynamic class and proceeding to - superclasses, looking for a concrete method matching the selector - of the message. If a concrete method is found, it is called. - - * If the class hierarchy contains no such method, a different method - is invoked on the class to give it an opportunity to dynamically - resolve the method; if taken, the process repeats, but skipping - this step the second time. - - * Otherwise, the global forwarding handler is invoked. On Darwin, - this is set up by CoreFoundation, and it follows a complicated - protocol which relies on NSInvocation. - - It is undefined behavior if the type signature of the method - implementation ultimately found is not compatible with the type - signature of the method that was used to statically type-check the - message send. This includes semantic annotations like ARC ownership - conventions and the ``noreturn`` attribute. Otherwise, there are no - semantic restrictions on what any particular method can do. - - - signature of the method implementation's - pr signature is not compatible with the signature at which the - method was invoked. - - -, in which case the runtime searches - the class hierarchy of the object, from most to least derived, - and calls the method - - -In Objective-C, every object has a class and every class has a -collection of methods. The high-level semantics are essentially -those - -.. nonsense ReST - - class is essentially a hashtable of selectors to - We propose a new attribute, ``@public``, that can adorn any - declaration not local to a function. For the purpose of standard - library development, even just parsing this attribute without - implementing semantics would be extremely useful in the near term. - -Basic Semantics -=============== - -``@public`` makes a declaration visible in code where the enclosing -module is imported. So, given this declaration in the ``Satchel`` -module:: - - @public struct Bag : ... { - ... - } - -We could write, in any other module, :: - - import Satchel - typealias SwingMe = Bag - -The difference from the status quo being that without ``@public`` on -the declaration of ``Bag``, the use of ``Bag`` above would be -ill-formed. - -Type-Checking -============= - -The types of all parameters and the return type of a func marked -``@public`` (including the implicit ``self`` of methods) must also be -``@public``. - -All parameters to a ``func`` marked ``@public`` (including the -implicit ``self`` of methods) must also be ``@public``:: - - struct X {} // not @public - @public struct Y {} - func f(_: X) {} // OK; also not @public - @public func g(_: Y) {} // OK; uses only @public types - @public func h(_: X, _: Y) {} // Ill-formed; non-public X in public signature - -A ``typealias`` marked ``@public`` must refer to a type marked -``@public``:: - - typealias XX = X // OK; not @public - @public typealias YY = Y // OK; Y is @public - @public typealias XXX = X // Ill-formed; public typealias refers to non-public type - -There is a straightforward and obvious rule for composing the -``@public``\ -ness of any compound type, including function types, -tuple types and instances of generic types: The compound type is -public if and only if all of the component types, are ``@public`` and -either defined in this module or re-exported from this module. - -Enums -===== - -The cases of an ``enum`` are ``@public`` if and only if the ``enum`` -is declared ``@public``. - -Derived Classes -=============== - -A method that overrides an ``@public`` method must be declared -``@public``, even if the enclosing class is non-``@public``. - -Protocols -========= - -A ``@public`` protocol can have ``@public`` and non-``@public`` -requirements. ``@public`` requirements can only be satisfied by -``@public`` declarations. Non-``@public`` requirements can be -satisfied by ``@public`` or non-``@public`` declarations. - -Conformances -============ - -The conformance of a type to a protocol is ``@public`` if that -conformance is part of an ``@public`` declaration. The program is -ill-formed if any declaration required to satisfy a ``@public`` -conformance is not also declared ``@public``.:: - - @public protocol P { - @public func f() { g() } - func g() - } - - struct X : P { // OK, X is not @public, so neither is its - func f() {} // conformance to P, and therefore f - func g() {} // can be non-@public - } - - protocol P1 {} - - @public struct Y : P1 {} // Y is @public so its - // conformance to P1 is, too. - - @public - extension Y : P { // This extension is @public, so - @public func f() {} // Y's conformance to P is also, and - func g() {} // thus f must be @public too - } - - protocol P2 {} - - extension Y : P2 {} // Y's conformance to P2 is non-@public - -.. Note:: It's our expectation that in the near term, and probably for - v1.0, non-``@public`` conformances on ``@public`` types will be - diagnosed as ill-formed/unsupported. - -A Related Naming Change -======================= - -The existing ``@exported`` attribute for imports should be renamed -``@public`` with no change in functionality. - -Future Directions -================= - -Some obvious directions to go in this feature space, which we are not -proposing today, but with which we tried to make this proposal -compatible: - -* non-``@public`` conformances -* file-private accessibility -* explicit non-``@public`` overrides, e.g. ``@!public`` - diff --git a/docs/proposals/OptimizerEffects.md b/docs/proposals/OptimizerEffects.md new file mode 100644 index 0000000000000..11cd0a49a4d49 --- /dev/null +++ b/docs/proposals/OptimizerEffects.md @@ -0,0 +1,914 @@ +Optimizer Effects: Summarizing and specifing function side effects +================================================================== + +> **note** +> +> This is a working document. Once we agree on the approach and +> terminology, this can move into docs/FunctionEffects.rst, and the +> codebase can be cleanup up a bit to reflect the consistent +> terminology. + +Introduction +------------ + +This document formalizes the effects that functions have on program +state for the purpose of facilitating compiler optimization. By modeling +more precise function effects, the optimizer can make more assumptions +leading to more aggressive transformation of the program. + +Function effects may be deduced by the compiler during program analyis. +However, in certain situations it is helpful to directly communicate +function effects to the compiler via function attributes or types. These +source level annotations may or may not be statically enforceable. + +This document specifies a comprehensive set of primitives and their +semantics. These primitives are the optimizer's interface to function +effects. They should be sufficient to express any analysis or annotation +supported by the compiler that express a function's effect on program +state. + +Within the optimizer, SILEffectsAnalysis deduces function effects +through analysis of SIL code. It's result directly maps to effects +primitives. + +Optimization of copy-on-Write data structures, such as Array, requires +guaranteed function effects that cannot be deduced through analysis of +SIL code. This necessitates a language for annotating functions. Some +annotations are obviously specific to CoW semantics and cannot be +defined in terms of general effects primitives: make\_unique, +preserve\_unique, and projects\_subobject. However, all other +annotations required for CoW optimization are really guarantees +regarding the program state that is affected by a CoW methods. These +annotations should map precisely to a set of effects primitives. + +For the sake of discussion, we use of Swift-level syntax for specifying +effects primitives. It may be debatable whether we actually want to +expose this syntax, but as explained above, some syntax will need to be +exposed to build optimizable CoW types. + +> **note** +> +> \[Andy\] There is a lot of subtlety involved in specifying or +> summarizing function effects. I want to first put forth an underlying +> model for reasoning about the effects' semantics, demonstrate that we +> can prove soundness in all the cases we care about for CoW and any +> other foreseeable purpose, then go back if needed and add sugary +> syntax and proper defaults for specifying the effects. I won't get +> hung up on the attributes being too fine grained or tricky to use. + +Effects Primitives +------------------ + +For the purpose of function effects, we identify program state that is +known reachable via an argument, versus some other unknown/unspecified +state. (Accessing a global variable is always unspecified state.) + +\[Andy\] We've given "global" a variety of meanings. I think we should +avoid that term unless specifically referring to global variables. + +There are some function level effects that are not specific to state: + +- allocs +- traps + +The effects on a particular state are: + +- read +- write +- capture +- release + +These should all be interpreted as effects that "may" happen. e.g. +maywrite, or mayretain. + +\[TODO\] Within the optimizer we sometimes refer to "retain" as an +effect. "capture" is really a more general term that encompasses any +retain\_value operation, so we'll likely standardize on that term. We +don't have a more general term for "release", which refers to any +release\_value operation. + +When referring to unspecified state, I will use the syntax +`@effects(no)`. When referring to state reachable via an +argument, `@no arg`. + +Naturally, we also need a syntax for associating effects with `self`. +That could easily be done by adding a @self\_effects attribute. + +In order to optimize bridged types, we need to add a `nonbridged` +predicate to the effects. The optimizer can then reason about a value's +bridged status within some scope and deduce more optimistic effects at a +call site. For now, we assume the predicate only applies to unspecified +state and that the bridged object is always self. That way we can denote +predicated effects as @nonbridged\_effects. + +In examples, @effects(argonly) means that there are no effects on +unspecified state. + +CoW Optimization Requirements +----------------------------- + +### Swift-level attributes proposal + +A copy-on-write (COW) type is implemented in terms of a struct and a set +of storage objects referenced by this struct. The set of storage objects +can further provide storage for subobjects.: + + class ArrayStorage { + func getElement(index: Int) -> T {} // Return a 'subobject'. + } + + struct Array { + var storage: ArrayStorage // Storage object + } + +In the following we will list a set of function attributes that can be +used to describe properties of methods of such a data structure to +facilitate optimization. + +A COW type implements value semantics by delaying the copy of storage of +the type until modification. + +An instance of a struct is in a uniqued state if changes to the set of +storage objects can only be observed by method calls on references to +the instance of the struct (versus by method calls on other instances). +Typically, one would implement this behavior by checking whether the +references to the storage objects are uniquely referenced and copying +the storage objects on modification if they are not. In the following we +refer to the memory holding the instance of the struct and the set of +storage objects as the self state. Non-self state below refers to the +state of the rest of the program not including the self state. + +`@make_unique` + +> A method marked `@make_unique` changes the state of the instance of +> the COW type (`self`) to the uniqued state. It must do so without +> changing or depending on non-self state or changing the self-state +> (other than the change to a uniqued state). It must be an idempotent +> operation.: +> +> struct Array { +> var storage: ArrayStorage +> +> @makeunique +> mutating func makeUnique() { +> if (isUniquelyReferenced(&storage)) +> return +> storage = storage.copy() +> } +> +> Note: In terms of low-level SIL attributes such a method will be +> marked:: +> +> @effects(argonly) +> @selfeffects(make_unique) +> func makeUnique() {} + +`@preserve_unique` + +> A method marked `@preserve_unique` must guarantee to not change the +> uniqueness state of `self` from a unique state to a not unique state. +> An example of a violation of this guarantee would be to store `self` +> in a global variable. The method must not return a storage object or +> address there-of that could be used to change the uniqueness state of +> `self`. An example of a violation of this guarantee would be a method +> that returns a storage object.: +> +> struct Array { +> var storage: ArrayStorage +> +> @preserve_unique +> mutating func replaceRange< +> C: CollectionType where C.Generator.Element == T +> >( +> subRange: Range, with newElements: C +> ) { ... } +> +> // We could also mark the following function as @preserve_unique +> // but we have an attribute for this function that better describes it +> // allowing for more optimization. (See @get_subobject) +> @preserve_unique +> func getElement(index: Int) -> T { +> return storage.elementAt(index) +> } +> } +> +> Note: In terms of low-level SIL attributes such a method will be +> marked:: +> +> @self_effects(preserve_unique, nocapture, norelease) +> func replaceRange<> {} + +`@get_subobject` + +> A method marked `@get_subobject` must fullfill all of +> `@preserve_unique`'s guarantees. Furthermore, it must return a +> 'subobject' that is stored by the set of storage objects or a value +> stored in the CoW struct itself. It must be guaranteed that the +> 'subobject' returned is kept alive as long the current value of the +> 'self' object is alive. Neither the self state nor the non-self state +> is changed and the method must not depend on non-self state.: +> +> struct Array { +> var storage: ArrayStorage +> var size : Int +> +> @get_subobject +> func getElement(index: Int) -> T { +> return storage.elementAt(index) +> } +> +> @get_subobject +> func getSize() -> Int { +> return size +> } +> +> Note: In terms of low-level SIL attributes such a method will be +> marked:: +> +> @effects(argonly) +> @selfeffects(preserve_unique, nowrite, nocapture, norelease, +> projects_subobject) +> func getElement(index: Int) -> T {} + +> **note** +> +> For the standard library's data types `@get_subobject` guarantees are +> too strong. An array can use an NSArray as its storage (it is in a +> bridged state) in which case we can't make assumptions on effects on +> non-self state. For this purpose we introduce a variant of the +> attribute above whose statement about global effects are predicated on +> the array being in a non-bridged state. + +`@get_subobject_non_bridged` + +> A method marked `@get_subobject` must fullfill all of +> `@preserve_unique`'s guarantees. Furthermore, it must return a +> 'subobject' that is stored by the set of storage objects or a value +> stored in the CoW struct itself. It must be guaranteed that the +> 'subobject' returned is kept alive as long the current value of the +> 'self' object is alive. The self state is not changed. The non-self +> state is not changed and the method must not depend on non-self state +> if the `self` is in a non-bridged state. In a bridged state the +> optimizer will assume that subsequent calls on the same 'self' object +> to return the same value and that consecutive calls are idempotent +> however it will not assume anything beyond this about effects on +> non-self state.: +> +> struct Array { +> var storage: BridgedArrayStorage +> var size : Int +> +> @get_subobject_non_bridged +> func getElement(index: Int) -> T { +> return storage.elementAt(index) +> } +> +> @get_subobject +> func getSize() -> Int { +> return size +> } +> +> Note: In terms of low-level SIL attributes such a method will be +> marked:: +> +> @nonbridged_effects(argonly) +> @selfeffects(preserve_unique, nowrite, nocapture, norelease, +> projects_subobject) +> func getElement(index: Int) -> T {} + +`@get_subobject_addr` + +> A method marked `@get_subobject_addr` must fullfill all of +> `@preserve_unique`'s guarantees. Furthermore, it must return the +> address of a 'subobject' that is stored by the set of storage objects. +> It is guaranteed that the 'subobject' at the address returned is kept +> alive as long the current value of the 'self' object is alive. Neither +> the self state nor the non-self state is changed and the method must +> not depend on non-self state.: +> +> struct Array { +> var storage: ArrayStorage +> +> @get_subobject_addr +> func getElementAddr(index: Int) -> UnsafeMutablePointer { +> return storage.elementAddrAt(index) +> } +> +> Note: In terms of low-level SIL attributes such a method will be +> marked:: +> +> @effects(argonly) +> @selfeffects(preserve_unique, nowrite, nocapture, norelease, +> projects_subobject_addr) +> func getElementAddr(index: Int) -> T {} + +`@initialize_subobject` + +> A method marked `@initialize_subobject` must fullfill all of +> `@preserve_unique`'s guarantees. The method must only store its +> arguments into *uninitialized* storage. The only effect to non-self +> state is the capture of the method's arguments.: +> +> struct Array { +> var storage: ArrayStorage +> +> @initialize_subobject +> func appendAssumingUniqueStorage(elt: T) { +> storage.append(elt) +> } +> } +> +> Note: In terms of low-level SIL attributes such a method will be +> marked:: +> +> @effects(argonly) +> @selfeffects(preserve_unique, nocapture, norelease) +> func appendElementAssumingUnique(@norelease @nowrite elt: T) {} + +> **note** +> +> \[arnold\] We would like to express something like `@set_subobject`, +> too. However, we probably want to delay this until we have a +> polymorphic effects type system. + +`@set_subobject` + +> A method marked `@set_subobject` must fullfill all of +> `@preserve_unique`'s guarantees. The method must only store its +> arguments into *initialized* storage. The only effect to non-self +> state is the capture of the method's arguments and the release of +> objects of the method arguments' types.: +> +> struct Array { +> var storage: ArrayStorage +> +> @set_subobject +> func setElement(elt: T, atIndex: Int) { +> storage.set(elt, atIndex) +> } +> } + +> **note** +> +> \[arnold\] As Andy points out, this would be best expressed using an +> effect type system. + +> Note: In terms of low-level SIL attributes such a method will be +> marked:: +> +> @effects(argonly, T.release) +> @selfeffects(preserve_unique, nocapture) +> func setElement(@nowrite e: T, index: Int) { +> } + +### Motivation + +Why do we need `makeunique`, `preserveunique`? + +The optimizer wants to hoist functions that make a COW type instance +unique out of loops. In order to do that it has to prove that uniqueness +is preserved by all operations in the loop. + +Marking methods as `makeunique`/`preserveunique` allows the optimizer to +reason about the behavior of the method calls. + +Example:: + + struct Array { + var storage: ArrayStorage + + @makeunique + func makeUnique() { + if (isUniquelyReferenced(&storage)) + return; + storage = storage.copy() + } + + @preserveunique + func getElementAddr(index: Int) -> UnsafeMutablePointer { + return storage.elementAddrAt(index) + } + + subscript(index: Int) -> UnsafeMutablePointer { + mutableAddressor { + makeUnique() + return getElementAddr(index) + } + } + } + +When the optimizer optimizes a loop:: + + func memset(inout A: [Int], value: Int) { + for i in 0 .. A.size { + A[i] = value + f() + } + } + +It will see the following calls because methods with attributes are not +inlined.: + + func memset(inout A: [Int], value: Int) { + for i in 0 .. A.size { + makeUnique(&A) + addr = getElementAddr(i, &A) + addr.memory = value + f() + } + } + +In order to hoist the 'makeUnique' call, the optimizer needs to be able +to reason that neither 'getElementAddr', nor the store to the address +returned can change the uniqueness state of 'A'. Furthermore, it knows +because 'A' is marked inout that in a program without inout violations f +cannot hold a reference to the object named by 'A' and therefore cannot +modify it. + +Why do we need `@get_subobject`, `@initialize_subobject`, and +`@set_subobject`? + +We want to be able to hoist `makeunique` calls when the array is not +identfied by a unique name.: + + class AClass { + var array: [Int] + } + + func copy(a : AClass, b : AClass) { + for i in min(a.size, b.size) { + a.array.append(b.array[i]) + } + } + +In such a case we would like to reason that:: + + = b.array[i] + +cannot changed the uniqueness of the instance of array 'a.array' +assuming 'a' !=== 'b'. We can do so because 'getElement' is marked +`@get_subobject` and so does not modify non-self state. + +Further we would like to reason that:: + + a.array.append + +cannot change the uniqueness state of the instance of array 'a.array' +accross iterations. We can conclude so because `appendAssumingUnique`'s +side-effects guarantee that no destructor can run - it's only +side-effect is that `tmp` is captured and initializes storage in the +array - these are the only side-effects according to +`@initialize_subobject`.: + + for i in 0 .. b.size { + // @get_subobject + tmp = getElement(b.array, i) + makeUnique(&a.array) + // @initialize_subobject + appendAssumingUnique(&a.array, tmp) + } + +We can construct a very similar example where we cannot hoist +makeUnique. If we replace 'getElement' with a 'setElement'. 'setElement' +will capture its argument and further releases an element of type T - +these are the only side-effects according to `@set_subobject`: + + @set_subobject + func setElement(e: T, index: Int) { + storage->setElement(e, index) + } + +Depending on 'T''s type a destructor can be invoked by the release of +'T'. The destructor can have arbitrary side-effects. Therefore, it is +not valid to hoist the makeUnique in the code without proving that 'T's +destructor cannot change the uniqueness state. This is trivial for +trivial types but requires a more sophisticated analysis for class types +(and in general cannot be disproved). In following example we can only +hoist makeUnique if we can prove that elt's, and elt2's destructor can't +change the uniqueness state of the arrays.: + + for i in 0 ..< min(a.size, b.size) { + makeUnique(&b.array) + setElement(&b.array, elt, i) + makeUnique(&a.array) + setElement(&a.array, elt2, i) + } + +In the following loop it is not safe to hoist the makeUnique(&a) call +even for trivial types. 'appendAssumingUnique' captures its argument 'a' +which forces a copy on 'a' on every iteration of the loop.: + + for i in 0 .. a.size { + makeUnique(&a) + setElement(&a, 0, i) + makeUnique(&b) + appendAssumingUnique(&b, a) + } + +To support this reasoning we need to know when a function captures its +arguments and when a function might release an object and of which type. + +`@get_subobject` and value-type behavior + +Furthermore, methods marked with `@get_subobject` will allow us to +remove redundant calls to read-only like methods on COW type instances +assuming we can prove that the instance is not changed in between them.: + + func f(a: [Int]) { + @get_subobject + count(a) + @get_subobject + count(a) + } + +Examples of Optimization Using Effects Primitives +------------------------------------------------- + +CoW optimization: \[Let's copy over examples from Arnold's proposal\] + +\[See the Copy-on-write proposal above\] + +String initialization: \[TBD\] + +User-Specified Effects, Syntax and Defaults +------------------------------------------- + +Mostly TBD. + +The optimizer can only take advantage of user-specified effects before +they have been inlined. Consequently, the optimizer initialy preserves +calls to annotated @effects() functions. After optimizing for effects +these functions can be inlined, dropping the effects information. + +Without special syntax, specifying a pure function would require: + + @effects(argonly) + func foo(@noread @nowrite arg) + +A shorthand, such as @effects(none) could easily be introduced. +Typically, this shouldn't be needed because the purity of a function can +probably be deduced from its argument types given that it has no effect +on unspecified state. i.e. If the function does not affect unspecific +state, and operates on "pure value types" (see below), the function is +pure. + +Specifying Effects for Generic Functions +---------------------------------------- + +Specifying literal function effects is not possible for functions with +generic arguments: + + struct MyContainer { + var t: T + func setElt(elt: T) { t = elt } + } + +With no knowledge of T.deinit() we must assume worst case. SIL effects +analysis following specialization can easily handle such a trivial +example. But there are two situations to be concerned about: + +1. Complicated CoW implementations defeat effects analysis. That is the + whole point of Arnold's proposal for user-specified CoW effects. +2. Eventually we will want to publish effects on generic functions + across resilience boundaries. + +Solving this requires a system for polymorphic effects. Language support +for polymorphic effects might look something like this: + + @effects(T.release) + func foo(t: T) { ... } + +This would mean that foo's unspecified effects are bounded by the +unspecified effects of T's deinitializer. The reality of designing +polymorphic effects will be much more complicated. + +A different approach would be to statically constrain effects on generic +types, protocol conformance, and closures. This wouldn't solve the +general problem, but could be a very useful tool for static enforcement. + +> **note** +> +> Examples of function effects systems: +> +> \[JoeG\] For example, the effect type system model in Koka +> () can handle exceptions, side effects on +> state, and heap capture in polymorphic contexts in a pretty elegant +> way. It's my hope that "throws" can provide a seed toward a full +> effects system like theirs. +> +> : A language with first-class effects. + +Purity +------ + +### Motivation for Pure Functions + +An important feature of Swift structs is that they can be defined such +that they have value semantics. The optimizer should then be able to +reason about these types with knowledge of those value semantics. This +in turn allows the optimizer to reason about function purity, which is a +powerful property. In particular, calls to pure functions can be hoisted +out of loops and combined with other calls taking the same arguments. +Pure functions also have no detrimental effect on optimizing the +surrounding code. + +For example: + + func bar(t: T) {...} + + func foo(t: T, N: Int) { + for _ in 1...N { + bar(t) + bar(t) + } + } + +With some knowledge of bar() and T can become: + + func foo(t: T, N: Int) { + bar(t) + } + +If our own implementation of value types, like Array, Set, and String +where annotated as know "pure values" and if their common operations are +known to comply with some low-level effects, then the optimizer could +infer more general purity of operations on those types. The optimizer +could then also reason about purity of operations on user defined types +composed from Arrays, Sets, and Strings. + +### "Pure" Value Types + +Conceptually, a pure value does not share state with another value. Any +trivial struct is automatically pure. Other structs can be declared pure +by the author. It then becomes the author's resonsibility to guarantee +value semantics. For instance, any stored reference into the heap must +either be to immutable data or protected by CoW. + +Since a pure value type can in practice share implementation state, we +need an enforcable definition of such types. More formally: + +- Copying or destroying a pure value cannot affect other + program state. +- Reading memory referenced from a pure value does not depend on other + program state. Writing memory referenced from a pure value cannot + affect other program state. + +The purity of functions that operate on these values, including their +own methods, must be deduced independently. + +From the optimizer perspective, there are two aspects of type purity +that fall out of the definition: + +(1) Side Effects of Copies + + Incrementing a reference count is not considered a side effect at + the level of value semantics. Destroying a pure value only destroys + objects that are part of the value's storage. This could be enforced + by prohibiting arbitrary code inside the storage deinitializer. + +(2) Aliasing + + Mutation of the pure value cannot affect program state apart from + that value, AND writing program state outside the value cannot + affect the pure value. + +\[Note\] Reference counts are exposed through the isUniquelyReferenced +API. Since copying a pure value can increase the reference of the +storage, strictly speaking, a pure function can have user-visible side +effects. We side step this issue by placing the burden on the user of +the isUniquelyReferenced API. The compiler only guarantees that the API +returns a non-unique reference count if there does happen to be an +aliasing reference after optimization, which the user cannot control. +The user must ensure that the program behaves identically in either case +apart from its performance characteristics. + +### Pure Value Types and SIL optimizations + +The benefit of having pure value types is that optimizations can treat +such types as if they were Swift value types, like struct. Member +functions of pure value types can be annotated with effects, like +`readnone` for `getElement`, even if the underlying implementation of +`getElement` reads memory from the type's storage. + +The compiler can do more optimistic optimizations for pure value types +without the need of sophisticated alias or escape analysis. + +Consider this example.: + + func add(arr: Array, i: Int) -> Int { + let e1 = arr[i] + unknownFunction() + let e2 = arr[i] + } + +This code is generated to something like: + + func add(arr: Array, i: Int) -> Int { + let e1 = getElement(i, arr) + unknownFunction() + let e2 = getElement(i, arr) + return e1 + e2 + } + +Now if the compiler can assume that Array is a pure value type and +`getElement` has a defined effect of `readnone`, it can CSE the two +calls. This is because the arguments, including the `arr` itself, are +the same for both calls. + +Even if `unknownFunction` modifies an array which references the same +storage as `arr`, CoW semantics will force `unknownFunction` to make a +copy of the storage and the storage of `arr` will not be modified. + +Pure value types can only considered pure on high-level SIL, before +effects and semantics functions are inlined. For an example see below. + +\[TBD\] Effects like `readnone` would have another impact on high-level +SIL than on low-level SIL. We have to decide how we want to handle this. + +### Recognizing Value Types + +A major difficulty in recognizing value types arises when those types +are implemented in terms of unsafe code with arbitrary side effects. +This is the crux of the difficulty in defining the CoW effects. +Consequently, communicating purity to the compiler will require some +function annotations and/or type constraints. + +A CoW type consits of a top-level value type, most likely a struct, and +a referenced storage, which may be shared between multiple instances of +the CoW type. + +\[TBD\] Is there any difference between a 'CoW type' and a 'pure value +type'? E.g. can there be CoW types which are not pure value types or +vice versa? + +The important thing for a pure value type is that all functions which +change the state are defined as mutating, even if they don't mutate the +top-level struct but only the referenced storage. + +> **note** +> +> For CoW data types this is required anyway, because any state-changing +> function will have to unique the storage and thus be able to replace +> the storage reference in the top-level struct. + +Let's assume we have a setElement function in Array.: + + mutating func setElement(i: Int, e: Element) { + storage[i] = e + } + +Let's replace the call to `unknownFunction` with a set of the i'th +element in our example. The mutating function forces the array to be +placed onto the stack and reloaded after the mutating function. This +lets the second `getElement` function get another array parameter which +prevents CSE of the two `getElement` calls. Shown in this swift-SIL +pseudo code: + + func add(var arr: Array, i: Int) -> Int { + let e1 = getElement(i, arr) + store arr to stack_array + setElement(i, 0, &stack_array) + let arr2 = load from stack_array + let e2 = getElement(i, arr2) // arr2 is another value than arr + return e1 + e2 + } + +Another important requirement for pure value types is that all +functions, which directly access the storage, are not inlined during +high-level SIL. Optimizations like code motion could move a store to the +storage over a `readnone getElement`.: + + func add(var arr: Array, i: Int) -> Int { + let e1 = getElement(i, arr) + store arr to stack_array + stack_array.storage[i] = 0 // (1) + let arr2 = load from stack_array // (2) + let e2 = getElement(i, arr2) // (3) + return e1 + e2 + } + +Store (1) and load (2) do not alias and (3) is defined as `readnone`. So +(1) could be moved over (3). + +Currently inlining is prevented in high-level SIL for all functions +which have an semantics or effect attribute. Therefore we could say that +the implementor of a pure value type has to define effects on all member +functions which eventually can access or modify the storage. + +To help the user to fulfill this contract, the compiler can check if +some effects annotations are missing. For this, the storage properties +of a pure value type should be annotated. The compiler can check if all +call graph paths from the type's member functions to storage accessing +functions contain at least one function with defined effects. Example: + + struct Array { + + @cow_storage var storage + + @effect(...) + func getElement() { return storage.get() } + + @effect(...) + func checkSubscript() { ... } + + subscript { get { // OK + checkSubscript() + return getElement() + } } + + func getSize() { + return storage.size() // Error! + } + } + +\[TBD\] What if a storage property is public. What if a non member +function accesses the storage. + +As discussed above, CoW types will often be generic, making the effects +of an operation on the CoW type dependent on the effects of destroying +an object of the element type. + +\[erik\] This is not the case if CoW types are always passed as +guaranteed to the effects functions. + +### Inferring Function Purity + +The optimizer can infer function purity by knowing that (1) the function +does not access unspecified state, (2) all arguments are pure values, +and (3) no calls are made into nonpure code. + +(1) The effects system described above already tells the optimizer via + analysis or annotation that the function does not access + unspecified state. +(2) Copying or destroying a pure value by definition has no impact on + other program state. The optimizer may either deduce this from the + type definition, or it may rely on a type constraint. +(3) Naturally, any calls within the function body must be + transitively pure. There is no need to check a calls to the storage + deinitializer, which should already be guaranteed pure by virtue + of (2). + +Mutability of a pure value should not affect the purity of functions +that operate on the value. An inout argument is semantically nothing +more than a copy of the value. + +\[Note\] Pure functions do not depend on or imply anything about the +reference counting effects: capture and release. Optimizations that +depend on reference count stability, like uniqueness hoisting, cannot +treat pure functions as side-effect free. + +> **note** +> +> \[Andy\] It may be possible to make some assumptions about +> immutability of `let` variables, which could lead to similar +> optimization. + +TODO: Need more clarity and examples + +Closures +-------- + +Mostly TBD. + +The optimizer does not currently have a way of statically determining or +enforcing effects of a function that takes a closure. We could introduce +attributes that statically enforce constraints. For example, and @pure +closure would only be permitted to close over pure values. + +> **note** +> +> \[Andy\] That is a fairly strict requirement, but not one that I know +> how to overcome. + +Thread Safety +------------- + +The Swift concurrency proposal refers to a `Copyable` type. A type must +be Copyable in order to pass it across threads via a `gateway`. The +definition of a Copyable type is equivalent to a "pure value". However, +it was also proposed that the programmer be able to annotate arbitrary +data types as Copyable even if they contain shared state as long as it +is protected via a mutex. However, such data types cannot be considered +pure by the optimizer. I instead propose that a separate constraint, +Synchronized, be attributed to shareable types that are not pure. An +object could be passed through a gateway either if it is a PureValue or +is Synchronized. + +Annotations for thread safety run into the same problems with generics +and closures. + +API and Resilience +------------------ + +Any type constraints, function effects, or closure attributes that we +introduce on public functions become part of the API. + +Naturally, there are resilience implications to user-specified effects. +Moving to a weaker set of declared effects is not resilient. + +Generally, a default-safe policy provides a much better user model from +some effects. For example, we could decide that functions cannot affect +unspecified state by default. If the user accesses globals, they then +need to annotate their function. However, default safety dictates that +any necessary annotations should be introduced before declaring API +stability. diff --git a/docs/proposals/OptimizerEffects.rst b/docs/proposals/OptimizerEffects.rst deleted file mode 100644 index 62cda039cbca8..0000000000000 --- a/docs/proposals/OptimizerEffects.rst +++ /dev/null @@ -1,923 +0,0 @@ -:orphan: - -.. OptimizerEffects: - -Optimizer Effects: Summarizing and specifing function side effects -================================================================== - -.. contents:: - -.. note:: - - This is a working document. Once we agree on the approach and - terminology, this can move into docs/FunctionEffects.rst, and the - codebase can be cleanup up a bit to reflect the consistent - terminology. - -Introduction ------------- - -This document formalizes the effects that functions have on program -state for the purpose of facilitating compiler optimization. By -modeling more precise function effects, the optimizer can make more -assumptions leading to more aggressive transformation of the program. - -Function effects may be deduced by the compiler during program -analyis. However, in certain situations it is helpful to directly -communicate function effects to the compiler via function attributes -or types. These source level annotations may or may not be statically -enforceable. - -This document specifies a comprehensive set of primitives and their -semantics. These primitives are the optimizer's interface to function -effects. They should be sufficient to express any analysis or -annotation supported by the compiler that express a function's effect -on program state. - -Within the optimizer, SILEffectsAnalysis deduces function effects -through analysis of SIL code. It's result directly maps to effects -primitives. - -Optimization of copy-on-Write data structures, such as Array, requires -guaranteed function effects that cannot be deduced through analysis of -SIL code. This necessitates a language for annotating functions. Some -annotations are obviously specific to CoW semantics and cannot be -defined in terms of general effects primitives: make_unique, -preserve_unique, and projects_subobject. However, all other annotations -required for CoW optimization are really guarantees regarding the -program state that is affected by a CoW methods. These annotations -should map precisely to a set of effects primitives. - -For the sake of discussion, we use of Swift-level syntax for -specifying effects primitives. It may be debatable whether we actually -want to expose this syntax, but as explained above, some syntax will -need to be exposed to build optimizable CoW types. - -.. note:: - - [Andy] There is a lot of subtlety involved in specifying or - summarizing function effects. I want to first put forth an - underlying model for reasoning about the effects' semantics, - demonstrate that we can prove soundness in all the cases we care - about for CoW and any other foreseeable purpose, then go back if - needed and add sugary syntax and proper defaults for specifying the - effects. I won't get hung up on the attributes being too fine - grained or tricky to use. - -Effects Primitives ------------------- - -For the purpose of function effects, we identify program state that is -known reachable via an argument, versus some other unknown/unspecified -state. (Accessing a global variable is always unspecified state.) - -[Andy] We've given "global" a variety of meanings. I think we should -avoid that term unless specifically referring to global variables. - -There are some function level effects that are not specific to state: - -- allocs -- traps - -The effects on a particular state are: - -- read -- write -- capture -- release - -These should all be interpreted as effects that "may" -happen. e.g. maywrite, or mayretain. - -[TODO] Within the optimizer we sometimes refer to "retain" as an -effect. "capture" is really a more general term that encompasses any -retain_value operation, so we'll likely standardize on that term. We -don't have a more general term for "release", which refers to any -release_value operation. - -When referring to unspecified state, I will use the syntax -``@effects(no)``. When referring to state reachable via an -argument, ``@no arg``. - -Naturally, we also need a syntax for associating effects with -``self``. That could easily be done by adding a @self_effects -attribute. - -In order to optimize bridged types, we need to add a ``nonbridged`` -predicate to the effects. The optimizer can then reason about a -value's bridged status within some scope and deduce more optimistic -effects at a call site. For now, we assume the predicate only applies -to unspecified state and that the bridged object is always self. That -way we can denote predicated effects as @nonbridged_effects. - -In examples, @effects(argonly) means that there are no effects on -unspecified state. - -CoW Optimization Requirements ------------------------------ - -Swift-level attributes proposal -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -A copy-on-write (COW) type is implemented in terms of a struct and a set of -storage objects referenced by this struct. The set of storage objects can -further provide storage for subobjects.:: - - class ArrayStorage { - func getElement(index: Int) -> T {} // Return a 'subobject'. - } - - struct Array { - var storage: ArrayStorage // Storage object - } - -In the following we will list a set of function attributes that can be used to -describe properties of methods of such a data structure to facilitate -optimization. - -A COW type implements value semantics by delaying the copy of storage of the -type until modification. - -An instance of a struct is in a uniqued state if changes to the set of storage -objects can only be observed by method calls on references to the instance of -the struct (versus by method calls on other instances). Typically, one would -implement this behavior by checking whether the references to the storage -objects are uniquely referenced and copying the storage objects on modification -if they are not. In the following we refer to the memory holding the instance -of the struct and the set of storage objects as the self state. Non-self state -below refers to the state of the rest of the program not including the self -state. - -``@make_unique`` - - A method marked ``@make_unique`` changes the state of the instance of the COW - type (``self``) to the uniqued state. It must do so without changing or - depending on non-self state or changing the self-state (other than the change - to a uniqued state). It must be an idempotent operation.:: - - struct Array { - var storage: ArrayStorage - - @makeunique - mutating func makeUnique() { - if (isUniquelyReferenced(&storage)) - return - storage = storage.copy() - } - - Note: In terms of low-level SIL attributes such a method will be marked::: - - @effects(argonly) - @selfeffects(make_unique) - func makeUnique() {} - -``@preserve_unique`` - - A method marked ``@preserve_unique`` must guarantee to not change the - uniqueness state of ``self`` from a unique state to a not unique state. An - example of a violation of this guarantee would be to store ``self`` in a - global variable. - The method must not return a storage object or address there-of that could be - used to change the uniqueness state of ``self``. An example of a violation of - this guarantee would be a method that returns a storage object.:: - - struct Array { - var storage: ArrayStorage - - @preserve_unique - mutating func replaceRange< - C: CollectionType where C.Generator.Element == T - >( - subRange: Range, with newElements: C - ) { ... } - - // We could also mark the following function as @preserve_unique - // but we have an attribute for this function that better describes it - // allowing for more optimization. (See @get_subobject) - @preserve_unique - func getElement(index: Int) -> T { - return storage.elementAt(index) - } - } - - Note: In terms of low-level SIL attributes such a method will be marked::: - - @self_effects(preserve_unique, nocapture, norelease) - func replaceRange<> {} - -``@get_subobject`` - - A method marked ``@get_subobject`` must fullfill all of ``@preserve_unique``'s - guarantees. Furthermore, it must return a 'subobject' that is stored by the - set of storage objects or a value stored in the CoW struct itself. It must be - guaranteed that the 'subobject' returned is kept alive as long the current - value of the 'self' object is alive. Neither the self state nor the non-self - state is changed and the method must not depend on non-self state.:: - - struct Array { - var storage: ArrayStorage - var size : Int - - @get_subobject - func getElement(index: Int) -> T { - return storage.elementAt(index) - } - - @get_subobject - func getSize() -> Int { - return size - } - - Note: In terms of low-level SIL attributes such a method will be marked::: - - @effects(argonly) - @selfeffects(preserve_unique, nowrite, nocapture, norelease, - projects_subobject) - func getElement(index: Int) -> T {} - -.. note:: - - For the standard library's data types ``@get_subobject`` guarantees are too - strong. An array can use an NSArray as its storage (it is in a bridged state) - in which case we can't make assumptions on effects on non-self state. For this - purpose we introduce a variant of the attribute above whose statement about - global effects are predicated on the array being in a non-bridged state. - -``@get_subobject_non_bridged`` - - A method marked ``@get_subobject`` must fullfill all of ``@preserve_unique``'s - guarantees. Furthermore, it must return a 'subobject' that is stored by the - set of storage objects or a value stored in the CoW struct itself. It must be - guaranteed that the 'subobject' returned is kept alive as long the current - value of the 'self' object is alive. The self state is not changed. The - non-self state is not changed and the method must not depend on non-self state - if the ``self`` is in a non-bridged state. In a bridged state the optimizer - will assume that subsequent calls on the same 'self' object to return the same - value and that consecutive calls are idempotent however it will not assume - anything beyond this about effects on non-self state.:: - - struct Array { - var storage: BridgedArrayStorage - var size : Int - - @get_subobject_non_bridged - func getElement(index: Int) -> T { - return storage.elementAt(index) - } - - @get_subobject - func getSize() -> Int { - return size - } - - Note: In terms of low-level SIL attributes such a method will be marked::: - - @nonbridged_effects(argonly) - @selfeffects(preserve_unique, nowrite, nocapture, norelease, - projects_subobject) - func getElement(index: Int) -> T {} - - -``@get_subobject_addr`` - - A method marked ``@get_subobject_addr`` must fullfill all of - ``@preserve_unique``'s guarantees. Furthermore, it must return the address of - a 'subobject' that is stored by the set of storage objects. It is guaranteed - that the 'subobject' at the address returned is kept alive as long the current - value of the 'self' object is alive. Neither the self state nor the non-self - state is changed and the method must not depend on non-self state.:: - - struct Array { - var storage: ArrayStorage - - @get_subobject_addr - func getElementAddr(index: Int) -> UnsafeMutablePointer { - return storage.elementAddrAt(index) - } - - Note: In terms of low-level SIL attributes such a method will be marked::: - - @effects(argonly) - @selfeffects(preserve_unique, nowrite, nocapture, norelease, - projects_subobject_addr) - func getElementAddr(index: Int) -> T {} - -``@initialize_subobject`` - - A method marked ``@initialize_subobject`` must fullfill all of - ``@preserve_unique``'s guarantees. The method must only store its arguments - into *uninitialized* storage. The only effect to non-self state is the capture - of the method's arguments.:: - - struct Array { - var storage: ArrayStorage - - @initialize_subobject - func appendAssumingUniqueStorage(elt: T) { - storage.append(elt) - } - } - - Note: In terms of low-level SIL attributes such a method will be marked::: - - @effects(argonly) - @selfeffects(preserve_unique, nocapture, norelease) - func appendElementAssumingUnique(@norelease @nowrite elt: T) {} - -.. note:: - - [arnold] We would like to express something like ``@set_subobject``, too. - However, we probably want to delay this until we have a polymorphic effects - type system. - -``@set_subobject`` - - A method marked ``@set_subobject`` must fullfill all of - ``@preserve_unique``'s guarantees. The method must only store its arguments - into *initialized* storage. The only effect to non-self state is the capture - of the method's arguments and the release of objects of the method arguments' - types.:: - - struct Array { - var storage: ArrayStorage - - @set_subobject - func setElement(elt: T, atIndex: Int) { - storage.set(elt, atIndex) - } - } - - -.. note:: - - [arnold] As Andy points out, this would be best expressed using an effect - type system. - - - Note: In terms of low-level SIL attributes such a method will be marked::: - - @effects(argonly, T.release) - @selfeffects(preserve_unique, nocapture) - func setElement(@nowrite e: T, index: Int) { - } - -Motivation -~~~~~~~~~~ - -Why do we need ``makeunique``, ``preserveunique``? - -The optimizer wants to hoist functions that make a COW type instance unique out -of loops. In order to do that it has to prove that uniqueness is preserved by -all operations in the loop. - -Marking methods as ``makeunique``/``preserveunique`` allows the optimizer to -reason about the behavior of the method calls. - -Example::: - - struct Array { - var storage: ArrayStorage - - @makeunique - func makeUnique() { - if (isUniquelyReferenced(&storage)) - return; - storage = storage.copy() - } - - @preserveunique - func getElementAddr(index: Int) -> UnsafeMutablePointer { - return storage.elementAddrAt(index) - } - - subscript(index: Int) -> UnsafeMutablePointer { - mutableAddressor { - makeUnique() - return getElementAddr(index) - } - } - } - -When the optimizer optimizes a loop::: - - func memset(inout A: [Int], value: Int) { - for i in 0 .. A.size { - A[i] = value - f() - } - } - -It will see the following calls because methods with attributes are not inlined.:: - - func memset(inout A: [Int], value: Int) { - for i in 0 .. A.size { - makeUnique(&A) - addr = getElementAddr(i, &A) - addr.memory = value - f() - } - } - -In order to hoist the 'makeUnique' call, the optimizer needs to be able to -reason that neither 'getElementAddr', nor the store to the address returned can -change the uniqueness state of 'A'. Furthermore, it knows because 'A' is marked -inout that in a program without inout violations f cannot hold a reference to -the object named by 'A' and therefore cannot modify it. - -Why do we need ``@get_subobject``, ``@initialize_subobject``, and -``@set_subobject``? - -We want to be able to hoist ``makeunique`` calls when the array is not identfied -by a unique name.:: - - class AClass { - var array: [Int] - } - - func copy(a : AClass, b : AClass) { - for i in min(a.size, b.size) { - a.array.append(b.array[i]) - } - } - -In such a case we would like to reason that::: - - = b.array[i] - -cannot changed the uniqueness of the instance of array 'a.array' assuming 'a' !=== 'b'. -We can do so because 'getElement' is marked ``@get_subobject`` and so does not -modify non-self state. - -Further we would like to reason that::: - - a.array.append - -cannot change the uniqueness state of the instance of array 'a.array' accross -iterations. We can conclude so because ``appendAssumingUnique``'s side-effects -guarantee that no destructor can run - it's only side-effect is that ``tmp`` -is captured and initializes storage in the array - these are the only -side-effects according to ``@initialize_subobject``.:: - - for i in 0 .. b.size { - // @get_subobject - tmp = getElement(b.array, i) - makeUnique(&a.array) - // @initialize_subobject - appendAssumingUnique(&a.array, tmp) - } - - -We can construct a very similar example where we cannot hoist makeUnique. If we -replace 'getElement' with a 'setElement'. 'setElement' will capture its argument -and further releases an element of type T - these are the only side-effects -according to ``@set_subobject``:: - - @set_subobject - func setElement(e: T, index: Int) { - storage->setElement(e, index) - } - -Depending on 'T''s type a destructor can be invoked by the release of 'T'. The -destructor can have arbitrary side-effects. Therefore, it is not valid to hoist -the makeUnique in the code without proving that 'T's destructor cannot change -the uniqueness state. This is trivial for trivial types but requires a more -sophisticated analysis for class types (and in general cannot be disproved). In -following example we can only hoist makeUnique if we can prove that elt's, and -elt2's destructor can't change the uniqueness state of the arrays.:: - - for i in 0 ..< min(a.size, b.size) { - makeUnique(&b.array) - setElement(&b.array, elt, i) - makeUnique(&a.array) - setElement(&a.array, elt2, i) - } - -In the following loop it is not safe to hoist the makeUnique(&a) -call even for trivial types. 'appendAssumingUnique' captures its argument 'a' -which forces a copy on 'a' on every iteration of the loop.:: - - for i in 0 .. a.size { - makeUnique(&a) - setElement(&a, 0, i) - makeUnique(&b) - appendAssumingUnique(&b, a) - } - -To support this reasoning we need to know when a function captures its -arguments and when a function might release an object and of which type. - -``@get_subobject`` and value-type behavior - -Furthermore, methods marked with ``@get_subobject`` will allow us to remove -redundant calls to read-only like methods on COW type instances assuming we can -prove that the instance is not changed in between them.:: - - func f(a: [Int]) { - @get_subobject - count(a) - @get_subobject - count(a) - } - - -Examples of Optimization Using Effects Primitives -------------------------------------------------- - -CoW optimization: [Let's copy over examples from Arnold's proposal] - -[See the Copy-on-write proposal above] - -String initialization: [TBD] - -User-Specified Effects, Syntax and Defaults -------------------------------------------- - -Mostly TBD. - -The optimizer can only take advantage of user-specified effects before -they have been inlined. Consequently, the optimizer initialy preserves -calls to annotated @effects() functions. After optimizing for effects -these functions can be inlined, dropping the effects information. - -Without special syntax, specifying a pure function would require:: - - @effects(argonly) - func foo(@noread @nowrite arg) - -A shorthand, such as @effects(none) could easily be -introduced. Typically, this shouldn't be needed because the purity of -a function can probably be deduced from its argument types given that -it has no effect on unspecified state. i.e. If the function does not -affect unspecific state, and operates on "pure value types" (see -below), the function is pure. - -Specifying Effects for Generic Functions ----------------------------------------- - -Specifying literal function effects is not possible for functions with -generic arguments:: - - struct MyContainer { - var t: T - func setElt(elt: T) { t = elt } - } - -With no knowledge of T.deinit() we must assume worst case. SIL effects -analysis following specialization can easily handle such a trivial -example. But there are two situations to be concerned about: - -1. Complicated CoW implementations defeat effects analysis. That is - the whole point of Arnold's proposal for user-specified CoW - effects. - -2. Eventually we will want to publish effects on generic functions - across resilience boundaries. - -Solving this requires a system for polymorphic effects. Language -support for polymorphic effects might look something like this:: - - @effects(T.release) - func foo(t: T) { ... } - -This would mean that foo's unspecified effects are bounded by the -unspecified effects of T's deinitializer. The reality of designing -polymorphic effects will be much more complicated. - -A different approach would be to statically constrain effects on -generic types, protocol conformance, and closures. This wouldn't solve -the general problem, but could be a very useful tool for static -enforcement. - -.. note:: Examples of function effects systems: - - [JoeG] For example, the effect type system model in Koka - (https://koka.codeplex.com) can handle exceptions, side - effects on state, and heap capture in polymorphic contexts in a - pretty elegant way. It's my hope that "throws" can provide a seed - toward a full effects system like theirs. - - http://www.eff-lang.org: A language with first-class effects. - - -Purity ------- - -Motivation for Pure Functions -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -An important feature of Swift structs is that they can be defined such -that they have value semantics. The optimizer should then be able to -reason about these types with knowledge of those value semantics. This -in turn allows the optimizer to reason about function purity, which is -a powerful property. In particular, calls to pure functions can be -hoisted out of loops and combined with other calls taking the same -arguments. Pure functions also have no detrimental effect on -optimizing the surrounding code. - -For example:: - - func bar(t: T) {...} - - func foo(t: T, N: Int) { - for _ in 1...N { - bar(t) - bar(t) - } - } - -With some knowledge of bar() and T can become:: - - func foo(t: T, N: Int) { - bar(t) - } - -If our own implementation of value types, like Array, Set, and String -where annotated as know "pure values" and if their common operations -are known to comply with some low-level effects, then the optimizer -could infer more general purity of operations on those types. The -optimizer could then also reason about purity of operations on user -defined types composed from Arrays, Sets, and Strings. - -"Pure" Value Types -~~~~~~~~~~~~~~~~~~ - -Conceptually, a pure value does not share state with another -value. Any trivial struct is automatically pure. Other structs can be -declared pure by the author. It then becomes the author's -resonsibility to guarantee value semantics. For instance, any stored -reference into the heap must either be to immutable data or protected -by CoW. - -Since a pure value type can in practice share implementation state, we -need an enforcable definition of such types. More formally: - -- Copying or destroying a pure value cannot affect other program - state. - -- Reading memory referenced from a pure value does not depend on other - program state. Writing memory referenced from a pure value cannot - affect other program state. - -The purity of functions that operate on these values, including their -own methods, must be deduced independently. - -From the optimizer perspective, there are two aspects of type purity -that fall out of the definition: - -(1) Side Effects of Copies - - Incrementing a reference count is not considered a side effect at - the level of value semantics. Destroying a pure value only - destroys objects that are part of the value's storage. This could - be enforced by prohibiting arbitrary code inside the storage deinitializer. - -(2) Aliasing - - Mutation of the pure value cannot affect program state apart from that value, - AND writing program state outside the value cannot affect the pure value. - -[Note] Reference counts are exposed through the isUniquelyReferenced -API. Since copying a pure value can increase the reference of the -storage, strictly speaking, a pure function can have user-visible side -effects. We side step this issue by placing the burden on the user of -the isUniquelyReferenced API. The compiler only guarantees that the -API returns a non-unique reference count if there does happen to be an -aliasing reference after optimization, which the user cannot -control. The user must ensure that the program behaves identically in -either case apart from its performance characteristics. - -Pure Value Types and SIL optimizations -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The benefit of having pure value types is that optimizations can treat such -types as if they were Swift value types, like struct. Member functions of pure -value types can be annotated with effects, like ``readnone`` for ``getElement``, -even if the underlying implementation of ``getElement`` reads memory from the -type's storage. - -The compiler can do more optimistic optimizations for pure value types without -the need of sophisticated alias or escape analysis. - -Consider this example.:: - - func add(arr: Array, i: Int) -> Int { - let e1 = arr[i] - unknownFunction() - let e2 = arr[i] - } - -This code is generated to something like:: - - func add(arr: Array, i: Int) -> Int { - let e1 = getElement(i, arr) - unknownFunction() - let e2 = getElement(i, arr) - return e1 + e2 - } - -Now if the compiler can assume that Array is a pure value type and ``getElement`` -has a defined effect of ``readnone``, it can CSE the two calls. This is because -the arguments, including the ``arr`` itself, are the same for both calls. - -Even if ``unknownFunction`` modifies an array which references the same storage -as ``arr``, CoW semantics will force ``unknownFunction`` to make a copy of the -storage and the storage of ``arr`` will not be modified. - -Pure value types can only considered pure on high-level SIL, before effects -and semantics functions are inlined. For an example see below. - -[TBD] Effects like ``readnone`` would have another impact on high-level SIL -than on low-level SIL. We have to decide how we want to handle this. - -Recognizing Value Types -~~~~~~~~~~~~~~~~~~~~~~~ - -A major difficulty in recognizing value types arises when those types -are implemented in terms of unsafe code with arbitrary side -effects. This is the crux of the difficulty in defining the CoW -effects. Consequently, communicating purity to the compiler will -require some function annotations and/or type constraints. - -A CoW type consits of a top-level value type, most likely a struct, and a -referenced storage, which may be shared between multiple instances of the CoW -type. - -[TBD] Is there any difference between a 'CoW type' and a 'pure value type'? -E.g. can there be CoW types which are not pure value types or vice versa? - -The important thing for a pure value type is that all functions which change -the state are defined as mutating, even if they don't mutate the top-level -struct but only the referenced storage. - -.. note:: - - For CoW data types this is required anyway, because any state-changing - function will have to unique the storage and thus be able to replace the - storage reference in the top-level struct. - -Let's assume we have a setElement function in Array.:: - - mutating func setElement(i: Int, e: Element) { - storage[i] = e - } - -Let's replace the call to ``unknownFunction`` with a set of the i'th element -in our example. -The mutating function forces the array to be placed onto the stack and reloaded -after the mutating function. This lets the second ``getElement`` function get -another array parameter which prevents CSE of the two ``getElement`` calls. -Shown in this swift-SIL pseudo code:: - - func add(var arr: Array, i: Int) -> Int { - let e1 = getElement(i, arr) - store arr to stack_array - setElement(i, 0, &stack_array) - let arr2 = load from stack_array - let e2 = getElement(i, arr2) // arr2 is another value than arr - return e1 + e2 - } - -Another important requirement for pure value types is that all functions, -which directly access the storage, are not inlined during high-level SIL. -Optimizations like code motion could move a store to the storage over a -``readnone getElement``.:: - - func add(var arr: Array, i: Int) -> Int { - let e1 = getElement(i, arr) - store arr to stack_array - stack_array.storage[i] = 0 // (1) - let arr2 = load from stack_array // (2) - let e2 = getElement(i, arr2) // (3) - return e1 + e2 - } - -Store (1) and load (2) do not alias and (3) is defined as ``readnone``. So (1) -could be moved over (3). - -Currently inlining is prevented in high-level SIL for all functions which -have an semantics or effect attribute. Therefore we could say that the -implementor of a pure value type has to define effects on all member functions -which eventually can access or modify the storage. - -To help the user to fulfill this contract, the compiler can check if some -effects annotations are missing. -For this, the storage properties of a pure value type should be annotated. -The compiler can check if all call graph paths -from the type's member functions to storage accessing functions contain at -least one function with defined effects. -Example:: - - struct Array { - - @cow_storage var storage - - @effect(...) - func getElement() { return storage.get() } - - @effect(...) - func checkSubscript() { ... } - - subscript { get { // OK - checkSubscript() - return getElement() - } } - - func getSize() { - return storage.size() // Error! - } - } - -[TBD] What if a storage property is public. What if a non member function -accesses the storage. - -As discussed above, CoW types will often be generic, making the -effects of an operation on the CoW type dependent on the effects of -destroying an object of the element type. - -[erik] This is not the case if CoW types are always passed as guaranteed -to the effects functions. - -Inferring Function Purity -~~~~~~~~~~~~~~~~~~~~~~~~~ - -The optimizer can infer function purity by knowing that (1) the -function does not access unspecified state, (2) all arguments are pure -values, and (3) no calls are made into nonpure code. - -(1) The effects system described above already tells the optimizer via - analysis or annotation that the function does not access - unspecified state. - -(2) Copying or destroying a pure value by definition has no impact on - other program state. The optimizer may either deduce this from the - type definition, or it may rely on a type constraint. - -(3) Naturally, any calls within the function body must be transitively - pure. There is no need to check a calls to the storage - deinitializer, which should already be guaranteed pure by virtue - of (2). - -Mutability of a pure value should not affect the purity of functions -that operate on the value. An inout argument is semantically nothing -more than a copy of the value. - -[Note] Pure functions do not depend on or imply anything about the -reference counting effects: capture and release. Optimizations that -depend on reference count stability, like uniqueness hoisting, cannot -treat pure functions as side-effect free. - -.. note:: - - [Andy] It may be possible to make some assumptions about - immutability of ``let`` variables, which could lead to similar - optimization. - -TODO: Need more clarity and examples - -Closures --------- - -Mostly TBD. - -The optimizer does not currently have a way of statically determining -or enforcing effects of a function that takes a closure. We could -introduce attributes that statically enforce constraints. For example, -and @pure closure would only be permitted to close over pure values. - -.. note:: - - [Andy] That is a fairly strict requirement, but not one that I know - how to overcome. - -Thread Safety -------------- - -The Swift concurrency proposal refers to a ``Copyable`` type. A type -must be Copyable in order to pass it across threads via a -``gateway``. The definition of a Copyable type is equivalent to a -"pure value". However, it was also proposed that the programmer be -able to annotate arbitrary data types as Copyable even if they contain -shared state as long as it is protected via a mutex. However, such -data types cannot be considered pure by the optimizer. I instead -propose that a separate constraint, Synchronized, be attributed to -shareable types that are not pure. An object could be passed through a -gateway either if it is a PureValue or is Synchronized. - -Annotations for thread safety run into the same problems with generics -and closures. - -API and Resilience ------------------- - -Any type constraints, function effects, or closure attributes that we -introduce on public functions become part of the API. - -Naturally, there are resilience implications to user-specified -effects. Moving to a weaker set of declared effects is not resilient. - -Generally, a default-safe policy provides a much better user model -from some effects. For example, we could decide that functions cannot -affect unspecified state by default. If the user accesses globals, -they then need to annotate their function. However, default safety -dictates that any necessary annotations should be introduced before -declaring API stability. diff --git a/docs/proposals/OptionSets.md b/docs/proposals/OptionSets.md new file mode 100644 index 0000000000000..3a773536ba2e9 --- /dev/null +++ b/docs/proposals/OptionSets.md @@ -0,0 +1,311 @@ +SUMMARY: Option sets should be structs of Bools, with a protocol to +provide bitwise-ish operations. + +Option Sets +=========== + +Option sets in C and ObjC are often represented using enums with +bit-pattern constants, as used in Cocoa's NS\_OPTIONS idiom. For +example: + + // ObjC + typedef NS_OPTIONS(NSUInteger, NSStringCompareOptions) { + NSCaseInsensitiveSearch = 1, + NSLiteralSearch = 2, + NSBackwardsSearch = 4, + NSAnchoredSearch = 8, + NSNumericSearch = 64, + NSDiacriticInsensitiveSearch = 128, + NSWidthInsensitiveSearch = 256, + NSForcedOrderingSearch = 512, + NSRegularExpressionSearch = 1024 + }; + +This approach doesn't map well to Swift's enums, which are intended to +be strict enumerations of states, or "sum types" to use the +type-theory-nerd term. An option set is more like a product type, and so +more naturally map to a struct of booleans: + + // Swift + struct NSStringCompareOptions { + var CaseInsensitiveSearch, + LiteralSearch, + BackwardsSearch, + AnchoredSearch, + NumericSearch, + DiacriticInsensitiveSearch, + WidthInsensitiveSearch, + ForcedOrderingSearch, + RegularExpressionSearch : Bool = false + } + +There are a few reasons this doesn't fly in C: + +- Boolean fields in C structs waste a byte by default. Option set + enums are compact. +- Bitfield ABI has historically been weird and unstable across + C implementations. Option set enums have a very concrete + binary representation. +- Prior to C99 it was difficult to use struct literals in expressions. +- It's useful to apply bitwise operations to option sets, which can't + be applied to C structs. +- Bitmasks also provide a natural way to express common option subsets + as constants, as in the `AllEdges` constants in the following + example: + + // ObjC + typedef NS_OPTIONS(unsigned long long, NSAlignmentOptions) { + NSAlignMinXInward = 1ULL << 0, + NSAlignMinYInward = 1ULL << 1, + NSAlignMaxXInward = 1ULL << 2, + NSAlignMaxYInward = 1ULL << 3, + NSAlignWidthInward = 1ULL << 4, + NSAlignHeightInward = 1ULL << 5, + + NSAlignMinXOutward = 1ULL << 8, + NSAlignMinYOutward = 1ULL << 9, + NSAlignMaxXOutward = 1ULL << 10, + NSAlignMaxYOutward = 1ULL << 11, + NSAlignWidthOutward = 1ULL << 12, + NSAlignHeightOutward = 1ULL << 13, + + NSAlignMinXNearest = 1ULL << 16, + NSAlignMinYNearest = 1ULL << 17, + NSAlignMaxXNearest = 1ULL << 18, + NSAlignMaxYNearest = 1ULL << 19, + NSAlignWidthNearest = 1ULL << 20, + NSAlignHeightNearest = 1ULL << 21, + + NSAlignRectFlipped = 1ULL << 63, // pass this if the rect is in a flipped coordinate system. This allows 0.5 to be treated in a visually consistent way. + + // convenience combinations + NSAlignAllEdgesInward = NSAlignMinXInward|NSAlignMaxXInward|NSAlignMinYInward|NSAlignMaxYInward, + NSAlignAllEdgesOutward = NSAlignMinXOutward|NSAlignMaxXOutward|NSAlignMinYOutward|NSAlignMaxYOutward, + NSAlignAllEdgesNearest = NSAlignMinXNearest|NSAlignMaxXNearest|NSAlignMinYNearest|NSAlignMaxYNearest, + }; + +However, we can address all of these issues in Swift. We should make the +theoretically correct struct-of-Bools representation also be the natural +and optimal way to express option sets. + +The 'OptionSet' Protocol +------------------------ + +One of the key features of option set enums is that, by using the +standard C bitwise operations, they provide easy and expressive +intersection, union, and negation of option sets. We can encapsulate +these capabilities into a protocol: + + // Swift + protocol OptionSet : Equatable { + // Set intersection + @infix func &(_:Self, _:Self) -> Self + @infix func &=(@inout _:Self, _:Self) + + // Set union + @infix func |(_:Self, _:Self) -> Self + @infix func |=(@inout _:Self, _:Self) + + // Set xor + @infix func ^(_:Self, _:Self) -> Self + @infix func ^=(@inout _:Self, _:Self) + + // Set negation + @prefix func ~(_:Self) -> Self + + // Are any options set? + func any() -> Bool + + // Are all options set? + func all() -> Bool + + // Are no options set? + func none() -> Bool + } + +The compiler can derive a default conformance for a struct whose +instance stored properties are all `Bool`: + + // Swift + struct NSStringCompareOptions : OptionSet { + var CaseInsensitiveSearch, + LiteralSearch, + BackwardsSearch, + AnchoredSearch, + NumericSearch, + DiacriticInsensitiveSearch, + WidthInsensitiveSearch, + ForcedOrderingSearch, + RegularExpressionSearch : Bool = false + } + + var a = NSStringCompareOptions(CaseInsensitiveSearch: true, + BackwardsSearch: true) + var b = NSStringCompareOptions(WidthInsensitiveSearch: true, + BackwardsSearch: true) + var c = a & b + (a & b).any() // => true + c == NSStringCompareOptions(BackwardsSearch: true) // => true + +Optimal layout of Bool fields in structs +---------------------------------------- + +Boolean fields should take up a single bit inside aggregates, avoiding +the need to mess with bitfields to get efficient layout. When used as +inout arguments, boolean fields packed into bits can go through +writeback buffers. + +Option Subsets +-------------- + +Option subsets can be expressed as static functions of the type. +(Ideally these would be static constants, if we had those.) For example: + + // Swift + struct NSAlignmentOptions : OptionSet { + var AlignMinXInward, + AlignMinYInward, + AlignMaxXInward, + AlignMaxYInward, + AlignWidthInward, + AlignHeightInward : Bool = false + + // convenience combinations + static func NSAlignAllEdgesInward() { + return NSAlignmentOptions(AlignMinXInward: true, + AlignMaxXInward: true, + AlignMinYInward: true, + AlignMaxYInward: true) + } + } + +Importing option sets from Cocoa +-------------------------------- + +When importing an NS\_OPTIONS declaration from Cocoa, we import it as an +OptionSet-conforming struct, with each single-bit member of the Cocoa +enum mapping to a Bool field of the struct with a default value of +`false`. Their IR-level layout places the fields at the correct bits to +be ABI-compatible with the C type. Multiple-bit constants are imported +as option subsets\_, mapping to static functions. + +*OPEN QUESTION*: What to do with bits that only appear as parts of +option subsets, as in: + + // ObjC + typedef NS_OPTIONS(unsigned, MyOptions) { + Foo = 0x01, + Bar = 0x03, // 0x02 | 0x01 + Bas = 0x05, // 0x04 | 0x01 + }; + +Areas for potential syntactic refinement +---------------------------------------- + +There are some things that are a bit awkward under this proposal which I +think are worthy of some examination. I don't have great solutions to +any of these issues off the top of my head. + +### Type and default value of option fields + +It's a bit boilerplate-ish to have to spell out the `: Bool = true` for +the set of fields: + + // Swift + struct MyOptions : OptionSet { + var Foo, + Bar, + Bas : Bool = false + } + +(though by comparison with C, it's still a net win, since the bitshifted +constants don't need to be manually spelled out and maintained. Is this +a big deal?) + +### Construction of option sets + +The implicit elementwise keyworded constructor for structs works +naturally for option set structs, except that it requires a bulky and +repetitive `: true` (or `: false`) after each keyword: + + // Swift + var myOptions = MyOptions(Foo: true, Bar: true) + +Some sort of shorthand for `keyword: true`/`keyword: false` would be +nice and would be generally useful beyond option sets, though I don't +have any awesome ideas of how that should look right now. + +### Nonuniformity of single options and option subsets + +Treating individual options and option subsets\_ differently disrupts +some of the elegance of the bitmask idiom. As static functions, option +subsets can't be combined freely in constructor calls like they can with +`|` in C. As instance stored properties, individual options must be +first constructed before bitwise operations can be applied to them. + + // ObjC + typedef NS_OPTIONS(unsigned, MyOptions) { + Foo = 0x01, + Bar = 0x02, + Bas = 0x04, + + Foobar = 0x03, + }; + + MyOptions x = Foobar | Bas; + + // Swift, under this proposal + struct MyOptions : OptionSet { + var Foo, Bar, Bas : Bool = false + + static func Foobar() -> MyOptions { + return MyOptions(Foo: true, Bar: true) + } + } + + var x: MyOptions = .Foobar() | MyOptions(Bas: true) + +This nonuniformity could potentially be addressed by introducing +additional implicit decls, such as adding implicit static properties +corresponding to each individual option: + + // Swift + struct MyOptions : OptionSet { + // Stored properties of instances + var Foo, Bar, Bas : Bool = false + + static func Foobar() -> MyOptions { + return MyOptions(Foo: true, Bar: true) + } + + // Implicitly-generated static properties? + static func Foo() -> MyOptions { return MyOptions(Foo: true) } + static func Bar() -> MyOptions { return MyOptions(Bar: true) } + static func Bas() -> MyOptions { return MyOptions(Bas: true) } + } + + var x: MyOptions = .Foobar() | .Bas() + +This is getting outside of strict protocol conformance derivation, +though. + +### Lack of static properties + +Static constant properties seem to me like a necessity to make option +subsets really acceptable to declare and use. This would be a much nicer +form of the above: + + // Swift + struct MyOptions : OptionSet { + // Stored properties of instances + var Foo, Bar, Bas : Bool = false + + static val Foobar = MyOptions(Foo: true, Bar: true) + + // Implicitly-generated static properties + static val Foo = MyOptions(Foo: true) + static val Bar = MyOptions(Bar: true) + static val Bas = MyOptions(Bas: true) + } + + var x: MyOptions = .Foobar | .Bas diff --git a/docs/proposals/OptionSets.rst b/docs/proposals/OptionSets.rst deleted file mode 100644 index fcc5b7a9bb01b..0000000000000 --- a/docs/proposals/OptionSets.rst +++ /dev/null @@ -1,319 +0,0 @@ -:orphan: - -SUMMARY: Option sets should be structs of Bools, with a protocol to provide -bitwise-ish operations. - -Option Sets -=========== - -Option sets in C and ObjC are often represented using enums with bit-pattern -constants, as used in Cocoa's NS_OPTIONS idiom. For example:: - - // ObjC - typedef NS_OPTIONS(NSUInteger, NSStringCompareOptions) { - NSCaseInsensitiveSearch = 1, - NSLiteralSearch = 2, - NSBackwardsSearch = 4, - NSAnchoredSearch = 8, - NSNumericSearch = 64, - NSDiacriticInsensitiveSearch = 128, - NSWidthInsensitiveSearch = 256, - NSForcedOrderingSearch = 512, - NSRegularExpressionSearch = 1024 - }; - -This approach doesn't map well to Swift's enums, which are intended to be -strict enumerations of states, or "sum types" to use the type-theory-nerd term. -An option set is more like a product type, and so more naturally map to a -struct of booleans:: - - // Swift - struct NSStringCompareOptions { - var CaseInsensitiveSearch, - LiteralSearch, - BackwardsSearch, - AnchoredSearch, - NumericSearch, - DiacriticInsensitiveSearch, - WidthInsensitiveSearch, - ForcedOrderingSearch, - RegularExpressionSearch : Bool = false - } - -There are a few reasons this doesn't fly in C: - -- Boolean fields in C structs waste a byte by default. Option set enums are - compact. -- Bitfield ABI has historically been weird and unstable across C - implementations. Option set enums have a very concrete binary representation. -- Prior to C99 it was difficult to use struct literals in expressions. -- It's useful to apply bitwise operations to option sets, which can't be - applied to C structs. -- Bitmasks also provide a natural way to express common option subsets as - constants, as in the ``AllEdges`` constants in the following example:: - - // ObjC - typedef NS_OPTIONS(unsigned long long, NSAlignmentOptions) { - NSAlignMinXInward = 1ULL << 0, - NSAlignMinYInward = 1ULL << 1, - NSAlignMaxXInward = 1ULL << 2, - NSAlignMaxYInward = 1ULL << 3, - NSAlignWidthInward = 1ULL << 4, - NSAlignHeightInward = 1ULL << 5, - - NSAlignMinXOutward = 1ULL << 8, - NSAlignMinYOutward = 1ULL << 9, - NSAlignMaxXOutward = 1ULL << 10, - NSAlignMaxYOutward = 1ULL << 11, - NSAlignWidthOutward = 1ULL << 12, - NSAlignHeightOutward = 1ULL << 13, - - NSAlignMinXNearest = 1ULL << 16, - NSAlignMinYNearest = 1ULL << 17, - NSAlignMaxXNearest = 1ULL << 18, - NSAlignMaxYNearest = 1ULL << 19, - NSAlignWidthNearest = 1ULL << 20, - NSAlignHeightNearest = 1ULL << 21, - - NSAlignRectFlipped = 1ULL << 63, // pass this if the rect is in a flipped coordinate system. This allows 0.5 to be treated in a visually consistent way. - - // convenience combinations - NSAlignAllEdgesInward = NSAlignMinXInward|NSAlignMaxXInward|NSAlignMinYInward|NSAlignMaxYInward, - NSAlignAllEdgesOutward = NSAlignMinXOutward|NSAlignMaxXOutward|NSAlignMinYOutward|NSAlignMaxYOutward, - NSAlignAllEdgesNearest = NSAlignMinXNearest|NSAlignMaxXNearest|NSAlignMinYNearest|NSAlignMaxYNearest, - }; - -However, we can address all of these issues in Swift. We should make the -theoretically correct struct-of-Bools representation also be the natural and -optimal way to express option sets. - -The 'OptionSet' Protocol ------------------------- - -One of the key features of option set enums is that, by using the standard C -bitwise operations, they provide easy and expressive intersection, union, and -negation of option sets. We can encapsulate these capabilities into a -protocol:: - - // Swift - protocol OptionSet : Equatable { - // Set intersection - @infix func &(_:Self, _:Self) -> Self - @infix func &=(@inout _:Self, _:Self) - - // Set union - @infix func |(_:Self, _:Self) -> Self - @infix func |=(@inout _:Self, _:Self) - - // Set xor - @infix func ^(_:Self, _:Self) -> Self - @infix func ^=(@inout _:Self, _:Self) - - // Set negation - @prefix func ~(_:Self) -> Self - - // Are any options set? - func any() -> Bool - - // Are all options set? - func all() -> Bool - - // Are no options set? - func none() -> Bool - } - -The compiler can derive a default conformance for a struct whose instance stored -properties are all ``Bool``:: - - // Swift - struct NSStringCompareOptions : OptionSet { - var CaseInsensitiveSearch, - LiteralSearch, - BackwardsSearch, - AnchoredSearch, - NumericSearch, - DiacriticInsensitiveSearch, - WidthInsensitiveSearch, - ForcedOrderingSearch, - RegularExpressionSearch : Bool = false - } - - var a = NSStringCompareOptions(CaseInsensitiveSearch: true, - BackwardsSearch: true) - var b = NSStringCompareOptions(WidthInsensitiveSearch: true, - BackwardsSearch: true) - var c = a & b - (a & b).any() // => true - c == NSStringCompareOptions(BackwardsSearch: true) // => true - -Optimal layout of Bool fields in structs ----------------------------------------- - -Boolean fields should take up a single bit inside aggregates, avoiding the need -to mess with bitfields to get efficient layout. When used as inout arguments, -boolean fields packed into bits can go through writeback buffers. - -Option Subsets --------------- - -Option subsets can be expressed as static functions of the type. -(Ideally these would be static constants, if we had those.) -For example:: - - // Swift - struct NSAlignmentOptions : OptionSet { - var AlignMinXInward, - AlignMinYInward, - AlignMaxXInward, - AlignMaxYInward, - AlignWidthInward, - AlignHeightInward : Bool = false - - // convenience combinations - static func NSAlignAllEdgesInward() { - return NSAlignmentOptions(AlignMinXInward: true, - AlignMaxXInward: true, - AlignMinYInward: true, - AlignMaxYInward: true) - } - } - -Importing option sets from Cocoa --------------------------------- - -When importing an NS_OPTIONS declaration from Cocoa, we import it as an -OptionSet-conforming struct, with each single-bit member of the Cocoa enum -mapping to a Bool field of the struct with a default value of ``false``. -Their IR-level layout places the fields -at the correct bits to be ABI-compatible with the C type. -Multiple-bit constants are imported as `option subsets`_, mapping to static -functions. - -*OPEN QUESTION*: What to do with bits that only appear as parts of option -subsets, as in:: - - // ObjC - typedef NS_OPTIONS(unsigned, MyOptions) { - Foo = 0x01, - Bar = 0x03, // 0x02 | 0x01 - Bas = 0x05, // 0x04 | 0x01 - }; - -Areas for potential syntactic refinement ----------------------------------------- - -There are some things that are a bit awkward under this proposal which -I think are worthy of some examination. I don't have great solutions to any of -these issues off the top of my head. - -Type and default value of option fields -``````````````````````````````````````` - -It's a bit boilerplate-ish to have to spell out the ``: Bool = true`` for the -set of fields:: - - // Swift - struct MyOptions : OptionSet { - var Foo, - Bar, - Bas : Bool = false - } - -(though by comparison with C, it's still a net win, since the bitshifted -constants don't need to be manually spelled out and maintained. Is this a big -deal?) - -Construction of option sets -``````````````````````````` - -The implicit elementwise keyworded constructor for structs works naturally for -option set structs, except that it requires a bulky and repetitive ``: true`` -(or ``: false``) after each keyword:: - - // Swift - var myOptions = MyOptions(Foo: true, Bar: true) - -Some sort of shorthand for ``keyword: true``/``keyword: false`` would be nice -and would be generally useful beyond option sets, though I don't have any -awesome ideas of how that should look right now. - -Nonuniformity of single options and option subsets -`````````````````````````````````````````````````` - -Treating individual options and `option subsets`_ differently disrupts some -of the elegance of the bitmask idiom. As static functions, option subsets can't -be combined freely in constructor calls like they can with ``|`` in C. As -instance stored properties, individual options must be first constructed before -bitwise operations can be applied to them. - -:: - - // ObjC - typedef NS_OPTIONS(unsigned, MyOptions) { - Foo = 0x01, - Bar = 0x02, - Bas = 0x04, - - Foobar = 0x03, - }; - - MyOptions x = Foobar | Bas; - -:: - - // Swift, under this proposal - struct MyOptions : OptionSet { - var Foo, Bar, Bas : Bool = false - - static func Foobar() -> MyOptions { - return MyOptions(Foo: true, Bar: true) - } - } - - var x: MyOptions = .Foobar() | MyOptions(Bas: true) - -This nonuniformity could potentially be addressed by introducing additional -implicit decls, such as adding implicit static properties corresponding to each -individual option:: - - // Swift - struct MyOptions : OptionSet { - // Stored properties of instances - var Foo, Bar, Bas : Bool = false - - static func Foobar() -> MyOptions { - return MyOptions(Foo: true, Bar: true) - } - - // Implicitly-generated static properties? - static func Foo() -> MyOptions { return MyOptions(Foo: true) } - static func Bar() -> MyOptions { return MyOptions(Bar: true) } - static func Bas() -> MyOptions { return MyOptions(Bas: true) } - } - - var x: MyOptions = .Foobar() | .Bas() - -This is getting outside of strict protocol conformance derivation, though. - -Lack of static properties -````````````````````````` - -Static constant properties seem to me like a necessity to make option subsets -really acceptable to declare and use. This would be a much nicer form of the -above:: - - // Swift - struct MyOptions : OptionSet { - // Stored properties of instances - var Foo, Bar, Bas : Bool = false - - static val Foobar = MyOptions(Foo: true, Bar: true) - - // Implicitly-generated static properties - static val Foo = MyOptions(Foo: true) - static val Bar = MyOptions(Bar: true) - static val Bas = MyOptions(Bas: true) - } - - var x: MyOptions = .Foobar | .Bas - diff --git a/docs/proposals/RemoteMirrors.md b/docs/proposals/RemoteMirrors.md new file mode 100644 index 0000000000000..8e9832e28be71 --- /dev/null +++ b/docs/proposals/RemoteMirrors.md @@ -0,0 +1,444 @@ +Remote mirrors proposal +======================= + +This proposal describes a new implementation for nominal type metadata +which will enable out-of-process heap inspection, intended for use by +debugging tools for detecting leaks and cycles. This implementation will +subsume the existing reflection support for Mirrors, enabling +out-of-process usage while also reducing generated binary size. + +Radars tracking this work: + +- rdar://problem/15617914 +- rdar://problem/17019505 +- rdar://problem/20771693 + +Goals and non-goals +------------------- + +We wish to do post-mortem debugging of memory allocations in a Swift +program. Debugging tools can already introspect the memory allocator to +identify all live memory allocations in the program's heap. + +If the compiler were to emit the necessary metadata, the layout of most +allocations can be ascertained, and in particular we can identify any +references inside the heap object. This metadata can be used together +with the core dump of a program to build a graph of objects. + +We have to be able to get all the necessary information without +executing any code in the address space of the target, since it may be +dead or otherwise in a funny state. + +In order to identify strong retain cycles, we need to know for each +reference if it is strong, weak, or unowned. + +We wish to be able to opt out of metadata selectively. For secrecy, we +might want to strip out field names, but keep metadata about which +fields contain references. For release builds, we might want to strip +out most of the field metadata altogether, except where explicitly +required for code that relies on reflection for functionality. + +It would be better if the new functionality subsumes some of the +existing metadata, instead of adding a whole new set of structures that +the compiler and runtime must keep in sync. + +While this should have zero runtime overhead when not in use, it is OK +if introspection requires some additional computation, especially if it +can be front-loaded or memoized. + +It is OK if in rare cases the metadata is not precise — for some +allocations, we might not be able to figure out the runtime type of the +contained value. Also we will not attempt to verify the "roots" of the +object graph by walking the stack for pointers. + +Types of heap allocations +------------------------- + +There are several types of heap allocations in Swift. We mostly concern +ourselves with class instances for now, but in the fullness of time we +would like to have accurate metadata for all heap allocations. + +### Swift class instances + +These have an isa pointer that points to a class metadata record. + +### Objective-C class instances + +These also have an isa pointer, but the class metadata record has the +Objective-C bit set. + +### Boxes + +These are used for heap-allocating mutable values captured in closures, +for indirect enum cases, and for ErrorType existential values. They have +an identifying isa pointer and reference count, but the isa pointer is +shared by all boxes and thus does not describe the heap layout of the +box. + +### Contexts + +The context for a thick function is laid out like a tuple consisting of +the captured values. Currently, the only aspect of the layout that is +needed by the runtime is knowledge of which captured values are heap +pointers. A unique isa pointer is created for each possible layout here. + +### Blocks + +Blocks are similar to contexts but have a common header and package the +function pointer and captured values in a single retainable heap object. + +### Metatypes + +Runtime-allocated metatypes will appear in the malloc heap. They +themselves cannot contain heap references though. + +### Opaque value buffers + +These come up when a value is too large to fit inside of an +existential's inline storage, for example. They do not have a header, so +we will not attempt to introspect them at first — eventually, we could +identify pointers to buffers where the existential is itself inside of a +heap-allocated object. + +Existing metadata +----------------- + +Swift already has a lot of reflective features and much of the +groundwork for this exists in some form or another, but each one is +lacking in at least one important respect. + +### Generic type metadata + +The isa pointer of an object points to a metadata record. For instances +of generic class types, the metadata is lazily instantiated from the +generic metadata template together with the concrete types that are +bound to generic parameters. + +Generic type metadata is instantiated for generic classes with live +instances, and for metatype records of value types which are explicitly +referenced from source. + +When the compiler needs to emit a generic type metadata record, it uses +one of several strategies depending on the type being referenced. For +concrete non-generic types, a direct call to a lazy accessor can be +generated. For bound generic types T<P1, ..., Pn>, we recursively +emit metadata references for the generic parameters Pn, then call the +getter for the bound type T. For archetypes — that is, generic type +parameters which are free variables in the function body being compiled +— the metadata is passed in as a value, so the compiler simply emits a +copy of that. + +Generic type metadata tells us the size of each heap allocation, but +does not by itself tell us the types of the fields or what references +they contain. + +### Mirrors and NominalTypeDescriptors + +The implementation of Mirrors uses runtime primitives which introspect +the fields of an opaque value by looking at the NominalTypeDescriptor +embedded in a type's metadata record. + +For structures and classes, the NominalTypeDescriptor contains a +function pointer which returns an array of field types. The function +pointer points to a "field type metadata function" emitted by the +compiler. This function emits metadata record references for each field +type and collects them in an array. Since the isa pointer of a class +instance points at an instantiated type, the field types of such a +NominalTypeDescriptor are also all concrete types. + +NominalTypeDescriptors record field names, in addition to types. Right +now, all of this information is stored together, without any way of +stripping it out. Also, NominalTypeDescriptors do not record whether a +reference is strong, weak or unowned, but that would be simple to fix. + +A bigger problem is that we have to call a function to lazily generate +the field type metadata. While a NominalTypeDescriptor for every +instantiated class type appears in a crashed process, the field types do +not, because only a call to the field type function will instantiate +them. + +### Objective-C instance variable metadata + +The Objective-C runtime keeps track of the types of instance variables +of classes, and there is enough information here to identify pointers in +instances of concrete types, however there's no support for generic +types. We could have generic type metadata instantiation also clone and +fill in templates for Objective-C instance variables, but this would add +a runtime cost to a feature that is primarily intended for debugging. + +### DWARF metadata + +IRGen emits some minimal amount of DWARF metadata for non-generic types, +but makes no attempt to describe generic type layout to the debugger in +this manner. + +However, DWARF has the advantage that it can be introspected without +running code, and stripped out. + +New field type metadata format +------------------------------ + +The main limitation of all of the above is either an inability to reason +about generic types, or the requirement to run code in the target. + +Suppose T is a generic type, and S is some set of substitutions. + +The compiler conceptually implements an operation G(T, S) which returns +a lazily-instantiated type descriptor for the given input parameters. +However, its really performing a partial evaluation G(T)(S), with the +"G(T)" part happening at compile time. + +Similarly, we can think of the field type access function as an +operation F(T, S) which returns the types of the fields of T, with T +again fixed at compile time. + +What we really want here is to build an "interpreter" — or really, a +parser for a simple serialized graph — which understands how to parse +uninstantiated generic metadata, keep track of substitutions, and +calculate field offsets, sizes, and locations of references. + +This "interpreter" has to be able to find metadata for leaf types "from +scratch", and calculate field sizes and offsets in the same way that +generic type metadata instantiation calculates object sizes. + +The "interpreter" will take the form of a library for understanding +field type metadata records and symbolic type references. This will be a +C++ library and it needs to support the following use cases: + +1. In-process reflection, for backing the current Mirrors in the + standard library +2. Out-of-process reflection, for heap debugging tools +3. Out-of-process reflection, for a new remote Mirrors feature in the + library (optional) + +The API will be somewhat similar to Mirrors as they are in the stdlib +today. + +The details are described below. + +### Symbolic type references + +Since we're operating on uninstantiated generic metadata, we need some +way to describe compositions of types. Instead of using metadata record +pointers, which are now insufficient, we use type references written in +a mini-language. + +A symbolic type reference is a recursive structure describing an +arbitrary Swift AST type in terms of nominal types, generic type +parameters, and compositions of them, such as tuple types. + +For each AST type, we can distinguish between the minimum information we +need to identify heap references therein, and the full type for +reflection. The former could be retained while the latter could be +stripped out in certain builds. + +We already have a very similar encoding — parameter type mangling in +SIL. It would be good to re-use this encoding, but for completeness, the +full format of a type reference is described below: + +1. **A built-in type reference.** Special tokens can be used to refer + to various built-in types that have runtime support. +2. **A concrete type reference.** This can either be a mangled name of + a type, or a GOT offset in the target. +3. **A heap reference.** This consists of: + - strong, weak or unowned + - (optional) a reference to the class type itself + +4. **A bound generic type.** This consists of: + - A concrete or built-in type reference + - A nested symbolic type reference for each generic parameter + +5. **A tuple type.** This consists of: + - A recursive sequence of symbolic type references. + +6. **A function type.** This consists of: + - A representation, + - (optional) input and output types + +7. **A protocol composition type.** This consists of: + - A flag indicating if any of the protocols are class-constrained, + which changes the representation + - The number of protocols in the composition + - (optional) references to all protocols in the composition + +8. **A metatype.** This consists of: + - (optional) a type reference to the instance type + - there's no required information — a metatype is always a single + pointer to a heap object which itself does not reference any + other heap objects. + +9. **An existential metatype.** This consists of: + - The number of protocols in the composition. + - (optional) type references to the protocol members. + +10. **A generic parameter.** Within the field types of a generic type, + references to generic parameters can appear. Generic parameters are + uniquely identifiable by an index here (and once we add nested + generic types, a depth). + +You can visualize type references as if they are written in an +S-expression format — but in reality, it would be serialized in a +compact binary form: + + (tuple_type + (bound_generic_type + (concrete_type "Array") + (concrete_type "Int")) + (bound_generic_type + (builtin_type "Optional") + (generic_type_parameter_type index=0))) + +We will provide a library of standalone routines for decoding, encoding +and manipulating symbolic type references. + +### Field type metadata records + +We introduce a new type of metadata, stored in its own section so that +it can be stripped out, called "field type metadata". For each nominal +type, we emit a record containing the following: + +1. the name of the nominal type, +2. the number of generic parameters, +3. type references, written in the mini-language above, for each of its + field types. +4. field names, if enabled. + +Field type metadata is linked together so that it can be looked up by +name, post-mortem by introspecting the core dump. + +We add a new field to the NominalTypeDescriptor to store a pointer to +field type metadata for this nominal type. In "new-style" +NominalTypeDescriptors that contain this field, the existing field type +function will point to a common field type function, defined in the +runtime, which instantiates the field type metadata. This allows for +backward compatibility with old code, if desired. + +### Field type metadata instantiation + +First, given an isa pointer in the target, we need to build the symbolic +type reference by walking backwards from instantiated to uninstantiated +metadata, collecting generic parameters. This operation is lazy, caching +the result for each isa pointer. + + enum SymbolicTypeReference { + case Concrete(String) + case BoundGeneric(String, [SymbolicTypeReference]) + case Tuple([SymbolicTypeReference]) + ... + } + + func getSymbolicTypeOfObject(isa: void*) -> SymbolicTypeReference + +Next, we define an "instantiation" operation, which takes a completely +substituted symbolic type reference, and returns a list of concrete +field types and offsets. + +This operation will need to recursively visit field metadata records and +keep track of generic parameter substitutions in order to correctly +calculate all field offsets and sizes. + +The result of instantiating metadata for each given +SymbolicTypeReference can be cached for faster lookup. + +This library has to be careful when following any pointers in the +target, to properly handle partially-initialized objects, runtime bugs +that led to memory corruption, or malicious code, without crashing or +exploiting the debugging tools. + + enum FieldLayout { + // the field contains a heap reference + case Strong, Weak, Unowned + // the field is an opaque binary blob, contents unknown. + case Opaque + // the field is a value type — look inside recursively. + case ValueType(indirect field: FieldDescriptor) + } + + struct FieldDescriptor { + let size: UInt + let align: UInt + let offset: UInt + let layout: FieldLayout + } + + func instantiateSymbolicType(ref: SymbolicTypeReference) -> [FieldTypeDescriptor] + +Field type metadata can have circular references — for example, consider +two classes which contain optionals of each other. In order to calculate +field offsets correctly, we need to break cycles when we know something +is a class type, and use a work-list algorithm instead of unbounded +recursion to ensure forward progress. + +### Enum type metadata + +For enums, the field metadata record will also need to contain enough +information about the spare bits and tag bits of the payload types that +we can at runtime determine the case of an enum and project the payload, +again without running code in the target. + +This will allow us to remove a pair of value witness functions generated +purely for reflection, since they don't seem to be performance-critical. + +### Closures + +For closure contexts and blocks, it would be nice to emit metadata, too. + +### Secrecy and release builds + +There are several levels of metadata we can choose to emit here: + +1. For code that requires runtime for functional purposes, or for the + standard library in debug builds, we can have a protocol conformance + or compiler flag enable unconditional emission of all metadata. +2. For system frameworks, we can omit field names and replace class + names with unique identifiers, but keep the type metadata to help + users debug memory leaks where framework classes are retaining + instances of user classes. +3. For release builds, we can strip out all the metadata except where + explicitly required in 1). + +This probably requires putting the required metadata in a different +section from the debug metadata. Perhaps field names should be separate +from symbolic type references too. + +### Performance + +Since the field type metadata instantiation only happens once per isa +pointer, mirrors will not suffer a performance impact beyond the initial +warm-up time. Once the field type descriptor has been constructed, +reflective access of fields will proceed as before. + +There might also be a marginal performance gain from removing all the +field type functions from the text segment, where they're currently +interspersed with other code, and replacing them with read only data +containing no relocations, which won't get paged in until needed. + +### Resilience + +We may choose to implement the new metadata facility after stabilizing +the ABI. In this case, we should front-load some engineering work on +NominalTypeDescriptors first, to make them more amenable to future +extension. + +We need to carefully review the new metadata format and make sure it is +flexible enough to support future language features, such as bound +generic existentials, which may further complicate heap layout. + +As described above, it is possible to introduce this change in a +backwards-compatible manner. We keep the field type function field in +the NominalTypeDescriptor, but for "new-style" records, set it to point +to a common function, defined in the runtime, which parses the new +metadata and returns an array of field types that can be used by old +clients. + +### Testing + +By transitioning mirrors to use the new metadata, existing tests can be +used to verify behavior. Additional tests can be developed to perform +various allocations and assert properties of the resulting object graph, +either from in- or out-of-process. + +If we go with the gradual approach where we have both field type +functions and field type metadata, we can also instantiate the former +and compare it against the result of invoking the latter, for all types +in the system, as a means of validating the field type metadata. diff --git a/docs/proposals/RemoteMirrors.rst b/docs/proposals/RemoteMirrors.rst deleted file mode 100644 index 66597fcded68c..0000000000000 --- a/docs/proposals/RemoteMirrors.rst +++ /dev/null @@ -1,471 +0,0 @@ -:orphan: - -Remote mirrors proposal -======================= - -.. contents:: - -This proposal describes a new implementation for nominal type metadata which -will enable out-of-process heap inspection, intended for use by debugging tools -for detecting leaks and cycles. This implementation will subsume the existing -reflection support for Mirrors, enabling out-of-process usage while also -reducing generated binary size. - -Radars tracking this work: - -- rdar://problem/15617914 -- rdar://problem/17019505 -- rdar://problem/20771693 - -Goals and non-goals -------------------- - -We wish to do post-mortem debugging of memory allocations in a Swift program. -Debugging tools can already introspect the memory allocator to identify all -live memory allocations in the program's heap. - -If the compiler were to emit the necessary metadata, the layout of most -allocations can be ascertained, and in particular we can identify any -references inside the heap object. This metadata can be used together with the -core dump of a program to build a graph of objects. - -We have to be able to get all the necessary information without executing any -code in the address space of the target, since it may be dead or otherwise in a -funny state. - -In order to identify strong retain cycles, we need to know for each reference -if it is strong, weak, or unowned. - -We wish to be able to opt out of metadata selectively. For secrecy, we might -want to strip out field names, but keep metadata about which fields contain -references. For release builds, we might want to strip out most of the field -metadata altogether, except where explicitly required for code that relies on -reflection for functionality. - -It would be better if the new functionality subsumes some of the existing -metadata, instead of adding a whole new set of structures that the compiler and -runtime must keep in sync. - -While this should have zero runtime overhead when not in use, it is OK if -introspection requires some additional computation, especially if it can be -front-loaded or memoized. - -It is OK if in rare cases the metadata is not precise — for some allocations, -we might not be able to figure out the runtime type of the contained value. -Also we will not attempt to verify the "roots" of the object graph by walking -the stack for pointers. - -Types of heap allocations -------------------------- - -There are several types of heap allocations in Swift. We mostly concern -ourselves with class instances for now, but in the fullness of time we would -like to have accurate metadata for all heap allocations. - -Swift class instances -~~~~~~~~~~~~~~~~~~~~~ - -These have an isa pointer that points to a class metadata record. - -Objective-C class instances -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -These also have an isa pointer, but the class metadata record has the -Objective-C bit set. - -Boxes -~~~~~ - -These are used for heap-allocating mutable values captured in closures, for -indirect enum cases, and for ErrorType existential values. They have an -identifying isa pointer and reference count, but the isa pointer is shared by -all boxes and thus does not describe the heap layout of the box. - -Contexts -~~~~~~~~ - -The context for a thick function is laid out like a tuple consisting of the -captured values. Currently, the only aspect of the layout that is needed by the -runtime is knowledge of which captured values are heap pointers. A unique isa -pointer is created for each possible layout here. - -Blocks -~~~~~~ - -Blocks are similar to contexts but have a common header and package the -function pointer and captured values in a single retainable heap object. - -Metatypes -~~~~~~~~~ - -Runtime-allocated metatypes will appear in the malloc heap. They themselves -cannot contain heap references though. - - -Opaque value buffers -~~~~~~~~~~~~~~~~~~~~ - -These come up when a value is too large to fit inside of an existential's -inline storage, for example. They do not have a header, so we will not attempt -to introspect them at first — eventually, we could identify pointers to buffers -where the existential is itself inside of a heap-allocated object. - -Existing metadata ------------------ - -Swift already has a lot of reflective features and much of the groundwork for -this exists in some form or another, but each one is lacking in at least one -important respect. - -Generic type metadata -~~~~~~~~~~~~~~~~~~~~~ - -The isa pointer of an object points to a metadata record. For instances of -generic class types, the metadata is lazily instantiated from the generic -metadata template together with the concrete types that are bound to generic -parameters. - -Generic type metadata is instantiated for generic classes with live instances, -and for metatype records of value types which are explicitly referenced from -source. - -When the compiler needs to emit a generic type metadata record, it uses one of -several strategies depending on the type being referenced. For concrete -non-generic types, a direct call to a lazy accessor can be generated. For bound -generic types T, we recursively emit metadata references for the -generic parameters Pn, then call the getter for the bound type T. For -archetypes — that is, generic type parameters which are free variables in the -function body being compiled — the metadata is passed in as a value, so the -compiler simply emits a copy of that. - -Generic type metadata tells us the size of each heap allocation, but does not -by itself tell us the types of the fields or what references they contain. - -Mirrors and NominalTypeDescriptors -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The implementation of Mirrors uses runtime primitives which introspect the -fields of an opaque value by looking at the NominalTypeDescriptor embedded in a -type's metadata record. - -For structures and classes, the NominalTypeDescriptor contains a function -pointer which returns an array of field types. The function pointer points to a -"field type metadata function" emitted by the compiler. This function emits -metadata record references for each field type and collects them in an array. -Since the isa pointer of a class instance points at an instantiated type, the -field types of such a NominalTypeDescriptor are also all concrete types. - -NominalTypeDescriptors record field names, in addition to types. Right now, all -of this information is stored together, without any way of stripping it out. -Also, NominalTypeDescriptors do not record whether a reference is strong, weak -or unowned, but that would be simple to fix. - -A bigger problem is that we have to call a function to lazily generate the -field type metadata. While a NominalTypeDescriptor for every instantiated class -type appears in a crashed process, the field types do not, because only a call -to the field type function will instantiate them. - -Objective-C instance variable metadata -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The Objective-C runtime keeps track of the types of instance variables of -classes, and there is enough information here to identify pointers in instances -of concrete types, however there's no support for generic types. We could have -generic type metadata instantiation also clone and fill in templates for -Objective-C instance variables, but this would add a runtime cost to a feature -that is primarily intended for debugging. - -DWARF metadata -~~~~~~~~~~~~~~ - -IRGen emits some minimal amount of DWARF metadata for non-generic types, but -makes no attempt to describe generic type layout to the debugger in this -manner. - -However, DWARF has the advantage that it can be introspected without running -code, and stripped out. - -New field type metadata format ------------------------------- - -The main limitation of all of the above is either an inability to reason about -generic types, or the requirement to run code in the target. - -Suppose T is a generic type, and S is some set of substitutions. - -The compiler conceptually implements an operation G(T, S) which returns a -lazily-instantiated type descriptor for the given input parameters. However, -its really performing a partial evaluation G(T)(S), with the "G(T)" part -happening at compile time. - -Similarly, we can think of the field type access function as an operation F(T, -S) which returns the types of the fields of T, with T again fixed at compile -time. - -What we really want here is to build an "interpreter" — or really, a parser for -a simple serialized graph — which understands how to parse uninstantiated -generic metadata, keep track of substitutions, and calculate field offsets, -sizes, and locations of references. - -This "interpreter" has to be able to find metadata for leaf types "from -scratch", and calculate field sizes and offsets in the same way that generic -type metadata instantiation calculates object sizes. - -The "interpreter" will take the form of a library for understanding field type -metadata records and symbolic type references. This will be a C++ library and -it needs to support the following use cases: - -#. In-process reflection, for backing the current Mirrors in the standard - library -#. Out-of-process reflection, for heap debugging tools -#. Out-of-process reflection, for a new remote Mirrors feature in the library - (optional) - -The API will be somewhat similar to Mirrors as they are in the stdlib today. - -The details are described below. - -Symbolic type references -~~~~~~~~~~~~~~~~~~~~~~~~ - -Since we're operating on uninstantiated generic metadata, we need some way to -describe compositions of types. Instead of using metadata record pointers, -which are now insufficient, we use type references written in a mini-language. - -A symbolic type reference is a recursive structure describing an arbitrary -Swift AST type in terms of nominal types, generic type parameters, and -compositions of them, such as tuple types. - -For each AST type, we can distinguish between the minimum information we need -to identify heap references therein, and the full type for reflection. The -former could be retained while the latter could be stripped out in certain -builds. - -We already have a very similar encoding — parameter type mangling in SIL. It -would be good to re-use this encoding, but for completeness, the full format of -a type reference is described below: - - -#. **A built-in type reference.** Special tokens can be used to refer to - various built-in types that have runtime support. - -#. **A concrete type reference.** This can either be a mangled name of a type, - or a GOT offset in the target. - -#. **A heap reference.** This consists of: - - - strong, weak or unowned - - (optional) a reference to the class type itself - -#. **A bound generic type.** This consists of: - - - A concrete or built-in type reference - - A nested symbolic type reference for each generic parameter - -#. **A tuple type.** This consists of: - - - A recursive sequence of symbolic type references. - -#. **A function type.** This consists of: - - - A representation, - - (optional) input and output types - -#. **A protocol composition type.** This consists of: - - - A flag indicating if any of the protocols are class-constrained, which - changes the representation - - The number of non-@objc protocols in the composition - - (optional) references to all protocols in the composition - -#. **A metatype.** This consists of: - - - (optional) a type reference to the instance type - - there's no required information — a metatype is always a single pointer to - a heap object which itself does not reference any other heap objects. - -#. **An existential metatype.** This consists of: - - - The number of protocols in the composition. - - (optional) type references to the protocol members. - -#. **A generic parameter.** Within the field types of a generic type, - references to generic parameters can appear. Generic parameters are uniquely - identifiable by an index here (and once we add nested generic types, a depth). - -You can visualize type references as if they are written in an S-expression -format — but in reality, it would be serialized in a compact binary form: - -:: - - (tuple_type - (bound_generic_type - (concrete_type "Array") - (concrete_type "Int")) - (bound_generic_type - (builtin_type "Optional") - (generic_type_parameter_type index=0))) - -We will provide a library of standalone routines for decoding, encoding and -manipulating symbolic type references. - -Field type metadata records -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -We introduce a new type of metadata, stored in its own section so that it can -be stripped out, called "field type metadata". For each nominal type, we emit a -record containing the following: - -#. the name of the nominal type, -#. the number of generic parameters, -#. type references, written in the mini-language above, for each of its field - types. -#. field names, if enabled. - -Field type metadata is linked together so that it can be looked up by name, -post-mortem by introspecting the core dump. - -We add a new field to the NominalTypeDescriptor to store a pointer to field -type metadata for this nominal type. In "new-style" NominalTypeDescriptors that -contain this field, the existing field type function will point to a common -field type function, defined in the runtime, which instantiates the field type -metadata. This allows for backward compatibility with old code, if desired. - -Field type metadata instantiation -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -First, given an isa pointer in the target, we need to build the symbolic type -reference by walking backwards from instantiated to uninstantiated metadata, -collecting generic parameters. This operation is lazy, caching the result for -each isa pointer. - -:: - - enum SymbolicTypeReference { - case Concrete(String) - case BoundGeneric(String, [SymbolicTypeReference]) - case Tuple([SymbolicTypeReference]) - ... - } - - func getSymbolicTypeOfObject(isa: void*) -> SymbolicTypeReference - -Next, we define an "instantiation" operation, which takes a completely -substituted symbolic type reference, and returns a list of concrete field types -and offsets. - -This operation will need to recursively visit field metadata records and keep -track of generic parameter substitutions in order to correctly calculate all -field offsets and sizes. - -The result of instantiating metadata for each given SymbolicTypeReference can -be cached for faster lookup. - -This library has to be careful when following any pointers in the target, to -properly handle partially-initialized objects, runtime bugs that led to memory -corruption, or malicious code, without crashing or exploiting the debugging -tools. - -:: - - enum FieldLayout { - // the field contains a heap reference - case Strong, Weak, Unowned - // the field is an opaque binary blob, contents unknown. - case Opaque - // the field is a value type — look inside recursively. - case ValueType(indirect field: FieldDescriptor) - } - - struct FieldDescriptor { - let size: UInt - let align: UInt - let offset: UInt - let layout: FieldLayout - } - - func instantiateSymbolicType(ref: SymbolicTypeReference) -> [FieldTypeDescriptor] - -Field type metadata can have circular references — for example, consider two -classes which contain optionals of each other. In order to calculate field -offsets correctly, we need to break cycles when we know something is a class -type, and use a work-list algorithm instead of unbounded recursion to ensure -forward progress. - -Enum type metadata -~~~~~~~~~~~~~~~~~~ - -For enums, the field metadata record will also need to contain enough -information about the spare bits and tag bits of the payload types that we can -at runtime determine the case of an enum and project the payload, again without -running code in the target. - -This will allow us to remove a pair of value witness functions generated purely -for reflection, since they don't seem to be performance-critical. - -Closures -~~~~~~~~ - -For closure contexts and blocks, it would be nice to emit metadata, too. - -Secrecy and release builds -~~~~~~~~~~~~~~~~~~~~~~~~~~ - -There are several levels of metadata we can choose to emit here: - -#. For code that requires runtime for functional purposes, or for the standard - library in debug builds, we can have a protocol conformance or compiler flag - enable unconditional emission of all metadata. -#. For system frameworks, we can omit field names and replace class names with - unique identifiers, but keep the type metadata to help users debug memory leaks - where framework classes are retaining instances of user classes. -#. For release builds, we can strip out all the metadata except where - explicitly required in 1). - -This probably requires putting the required metadata in a different section -from the debug metadata. Perhaps field names should be separate from symbolic -type references too. - -Performance -~~~~~~~~~~~ - -Since the field type metadata instantiation only happens once per isa pointer, -mirrors will not suffer a performance impact beyond the initial warm-up time. -Once the field type descriptor has been constructed, reflective access of -fields will proceed as before. - -There might also be a marginal performance gain from removing all the field -type functions from the text segment, where they're currently interspersed with -other code, and replacing them with read only data containing no relocations, -which won't get paged in until needed. - -Resilience -~~~~~~~~~~ - -We may choose to implement the new metadata facility after stabilizing the ABI. -In this case, we should front-load some engineering work on -NominalTypeDescriptors first, to make them more amenable to future extension. - -We need to carefully review the new metadata format and make sure it is -flexible enough to support future language features, such as bound generic -existentials, which may further complicate heap layout. - -As described above, it is possible to introduce this change in a -backwards-compatible manner. We keep the field type function field in the -NominalTypeDescriptor, but for "new-style" records, set it to point to a common -function, defined in the runtime, which parses the new metadata and returns an -array of field types that can be used by old clients. - -Testing -~~~~~~~ - -By transitioning mirrors to use the new metadata, existing tests can be used to -verify behavior. Additional tests can be developed to perform various -allocations and assert properties of the resulting object graph, either from -in- or out-of-process. - -If we go with the gradual approach where we have both field type functions and -field type metadata, we can also instantiate the former and compare it against -the result of invoking the latter, for all types in the system, as a means of -validating the field type metadata. - diff --git a/docs/proposals/TypeState.md b/docs/proposals/TypeState.md new file mode 100644 index 0000000000000..08a757e7c16d3 --- /dev/null +++ b/docs/proposals/TypeState.md @@ -0,0 +1,153 @@ +General Type State Notes +======================== + +Immutability +------------ + +Using Typestate to control immutability requires recursive immutability +propagation (just like sending a value in a message does a recursive +deep copy). This brings up interesting questions: + +1. should types be able to opt-in or out of Immutabilizability? +2. It seems that 'int' shouldn't be bloated by tracking the possibility + of immutabilizability. +3. We can reserve a bit in the object header for reference types to + indicate "has become immutable". +4. If a type opts-out of immutabilization (either explicitly + or implicitly) then a recursive type derived from it can only be + immutabilized if the type is explicitly marked immutable. For + example, you could only turn a struct immutable if it contained + "const int's"? Or is this really only true for reference types? It + seems that the immutability of a value-type element can follow the + immutability of the containing object. Array slices need a pointer + to the containing object for more than just the refcount it seems. + +Typestate + GC + ARC +-------------------- + +A random email from Mike Ferris. DVTInvalidation models a type state, +one which requires recursive transitive propagation just like immutable +does: + +"For what it is worth, Xcode 4 has a general DVTInvalidation protocol +that many of our objects adopt. This was a hard-won lesson dealing with +GC where just because something is ready to be collected does not mean +it will be immediately. + +We use this to clean up held resources and as a statement of intent that +this object is now "done". Many of our objects that conform to this +protocol also assert validity in key external entry points to attempt to +enforce that once they're invalid, no one should be talking to them. + +In a couple cases we have found single-ownership to be insufficient and, +in those cases, we do have, essentially, ref-counting of validity. But +in the vast majority of cases, there is a single owner who \_should\_ be +controlling the useful lifetime of these objects. And anyone else +keeping them alive after that useful lifetime is basically in error (and +is in a position to be caught by our validity assertions.) + +At some point I am sure we'll be switching to ARC and, as we do, the +forcing function that caused us to adopt the DVTInvalidation pattern may +fall by the wayside (i.e. the arbitrary latency between ready to be +collected and collected). But I doubt we would consider not having the +protocol as we do this. It has been useful in many ways to formalize +this notion if only because it forces more rigorous consideration of +ownership models and gives us a pragmatic way to enforce them. + +The one thing that has been a challenge is that adoption of +DVTInvalidation is somewhat viral. If you own an object that in +invalidate-able, then you pretty much have to be invalidate-able +yourself (or have an equivalent guaranteed trigger to be sure you'll +eventually invalidate the object)... Over time, more and more of our +classes wind up adopting this protocol. I am not sure that's a bad +thing, but it has been an observed effect of having this pattern." + +Plaid Language notes +-------------------- + + aka + +This paper uses the hybrid dynamic/static approach I chatted to Ted +about (which attaches dynamic tags to values, which the optimizer then +tries to remove). This moves the approach from "crazy theory" to "has at +least been implemented somewhere once": + + +It allows typestate changes to change representation. It sounds to me +like conjoined discriminated unions + type state. + +Cute typestate example: the state transition from egg, to caterpillar, +to pupae, to butterfly. + +It only allows data types with finite/enumerable typestates. + +It defines typestates with syntax that looks like it is defining types: + + state File { + val filename; + } + + state OpenFile case of File = { + val filePtr; + method read() { ... } + method close() { this <- ClosedFile; } + } + + state ClosedFile case of File { + method open() { this <- OpenFile; } + } + +Makes it really seem like a discriminated union. The stated reason to do +this is to avoid having "null pointers" and other invalid data around +when in a state where it is not valid. It seems that another reasonable +approach would be to tag data members as only being valid in some +states. Both have tradeoffs. Doing either of them would be a great way +to avoid having to declare stuff "optional/?" just because of typestate, +and even permits other types that don't have a handy sentinel. It is +still useful to define unconditional data, and still useful to allow +size-optimization by deriving state from a field ("-1 is a closed file +state" - at least if we don't have good integer size bounds, which we do +want anyway). + +It strikes me that typestate declarations themselves (e.g. a type can be +in the "open" or "closed" state) should be independently declared from +types and should have the same sort of visibility controls as types. I +should be able to declare a protocol/java interface along the lines of: + + protocol fileproto { + open(...) closed; + close(...) opened; + } + +using "public" closed/opened states. Insert fragility concerns here. + +It supports multidimensional typestate, where a class can transition in +multiple dimensions without having to manually manage a matrix of +states. This seems particularly useful in cases where you have +inheritance. A base class may define its own set of states. A derived +class will have those states, plus additional dimensions if they wanted. +For example, an NSView could be visible or not, while a NSButton derived +class could be Normal or Pressed Down, etc. + +Generics: "mechanisms like type parameterization need to be duplicated +for typestate, so that we can talk not only about a list of files, but +also about a list of *open* files". + +You should be allowed to declare typestate transitions on "self" any any +by-ref arguments/ret values on functions. In Plaid syntax: + + public void open() [ClosedFile>>OpenFile] + +should be a precondition that 'self' starts out in the ClosedFile state +and a postcondition that it ends up in the OpenFile state. The +implementation could be checked against this contract. + +Their onward2009 paper contains the usual set of aliasing restrictions +and conflation of immutable with something-not-typestate that I come to +expect from the field. + +Their examples remind me that discriminated unions should be allowed to +have a 'base class': data that is common and available across all the +slices. Changing to another slice should not change this stuff. + +'instate' is the keyword they choose to use for a dynamic state test. diff --git a/docs/proposals/TypeState.rst b/docs/proposals/TypeState.rst deleted file mode 100644 index 6e3e57da52dd5..0000000000000 --- a/docs/proposals/TypeState.rst +++ /dev/null @@ -1,154 +0,0 @@ -:orphan: - -.. @raise litre.TestsAreMissing - -General Type State Notes -======================== - -Immutability ------------- - -Using Typestate to control immutability requires recursive immutability -propagation (just like sending a value in a message does a recursive deep copy). -This brings up interesting questions: - -1. should types be able to opt-in or out of Immutabilizability? - -2. It seems that 'int' shouldn't be bloated by tracking the possibility of - immutabilizability. - -3. We can reserve a bit in the object header for reference types to indicate - "has become immutable". - -4. If a type opts-out of immutabilization (either explicitly or implicitly) then - a recursive type derived from it can only be immutabilized if the type is - explicitly marked immutable. For example, you could only turn a struct - immutable if it contained "const int's"? Or is this really only true for - reference types? It seems that the immutability of a value-type element can - follow the immutability of the containing object. Array slices need a - pointer to the containing object for more than just the refcount it seems. - -Typestate + GC + ARC --------------------- - -A random email from Mike Ferris. DVTInvalidation models a type state, one which -requires recursive transitive propagation just like immutable does: - -"For what it is worth, Xcode 4 has a general DVTInvalidation protocol that many -of our objects adopt. This was a hard-won lesson dealing with GC where just -because something is ready to be collected does not mean it will be immediately. - -We use this to clean up held resources and as a statement of intent that this -object is now "done". Many of our objects that conform to this protocol also -assert validity in key external entry points to attempt to enforce that once -they're invalid, no one should be talking to them. - -In a couple cases we have found single-ownership to be insufficient and, in -those cases, we do have, essentially, ref-counting of validity. But in the vast -majority of cases, there is a single owner who _should_ be controlling the -useful lifetime of these objects. And anyone else keeping them alive after that -useful lifetime is basically in error (and is in a position to be caught by our -validity assertions.) - -At some point I am sure we'll be switching to ARC and, as we do, the forcing -function that caused us to adopt the DVTInvalidation pattern may fall by the -wayside (i.e. the arbitrary latency between ready to be collected and -collected). But I doubt we would consider not having the protocol as we do -this. It has been useful in many ways to formalize this notion if only because -it forces more rigorous consideration of ownership models and gives us a -pragmatic way to enforce them. - -The one thing that has been a challenge is that adoption of DVTInvalidation is -somewhat viral. If you own an object that in invalidate-able, then you pretty -much have to be invalidate-able yourself (or have an equivalent guaranteed -trigger to be sure you'll eventually invalidate the object)... Over time, more -and more of our classes wind up adopting this protocol. I am not sure that's a -bad thing, but it has been an observed effect of having this pattern." - -Plaid Language notes --------------------- - -http://plaid-lang.org/ aka http://www.cs.cmu.edu/~aldrich/plaid/ - -This paper uses the hybrid dynamic/static approach I chatted to Ted about (which -attaches dynamic tags to values, which the optimizer then tries to remove). This -moves the approach from "crazy theory" to "has at least been implemented -somewhere once": http://www.cs.cmu.edu/~aldrich/papers/plaid-oopsla11.pdf - -It allows typestate changes to change representation. It sounds to me like -conjoined discriminated unions + type state. - -Cute typestate example: the state transition from egg, to caterpillar, to pupae, -to butterfly. - -It only allows data types with finite/enumerable typestates. - -It defines typestates with syntax that looks like it is defining types:: - - state File { - val filename; - } - - state OpenFile case of File = { - val filePtr; - method read() { ... } - method close() { this <- ClosedFile; } - } - - state ClosedFile case of File { - method open() { this <- OpenFile; } - } - -Makes it really seem like a discriminated union. The stated reason to do this -is to avoid having "null pointers" and other invalid data around when in a state -where it is not valid. It seems that another reasonable approach would be to -tag data members as only being valid in some states. Both have tradeoffs. -Doing either of them would be a great way to avoid having to declare stuff -"optional/?" just because of typestate, and even permits other types that don't -have a handy sentinel. It is still useful to define unconditional data, and -still useful to allow size-optimization by deriving state from a field ("-1 is a -closed file state" - at least if we don't have good integer size bounds, which -we do want anyway). - -It strikes me that typestate declarations themselves (e.g. a type can be in the -"open" or "closed" state) should be independently declared from types and should -have the same sort of visibility controls as types. I should be able to declare -a protocol/java interface along the lines of:: - - protocol fileproto { - open(...) closed; - close(...) opened; - } - -using "public" closed/opened states. Insert fragility concerns here. - -It supports multidimensional typestate, where a class can transition in multiple -dimensions without having to manually manage a matrix of states. This seems -particularly useful in cases where you have inheritance. A base class may -define its own set of states. A derived class will have those states, plus -additional dimensions if they wanted. For example, an NSView could be visible -or not, while a NSButton derived class could be Normal or Pressed Down, etc. - -Generics: "mechanisms like type parameterization need to be duplicated for -typestate, so that we can talk not only about a list of files, but also about a -list of *open* files". - - -You should be allowed to declare typestate transitions on "self" any any by-ref -arguments/ret values on functions. In Plaid syntax:: - - public void open() [ClosedFile>>OpenFile] - -should be a precondition that 'self' starts out in the ClosedFile state and a -postcondition that it ends up in the OpenFile state. The implementation could -be checked against this contract. - -Their onward2009 paper contains the usual set of aliasing restrictions and -conflation of immutable with something-not-typestate that I come to expect from -the field. - -Their examples remind me that discriminated unions should be allowed to have a -'base class': data that is common and available across all the slices. Changing -to another slice should not change this stuff. - -'instate' is the keyword they choose to use for a dynamic state test. diff --git a/docs/proposals/ValueSemantics.md b/docs/proposals/ValueSemantics.md new file mode 100644 index 0000000000000..8624b1957ad86 --- /dev/null +++ b/docs/proposals/ValueSemantics.md @@ -0,0 +1,298 @@ +Value Semantics in Swift +======================== + +Abstract + +: Swift is the first language to take Generic Programming seriously + that also has both value and reference types. The (sometimes subtle) + differences between the behaviors of value and reference types + create unique challenges for generic programs that we have not + yet addressed. This paper surveys related problems and explores some + possible solutions. + +Definitions +----------- + +I propose the following definitions of "value semantics" and "reference +semantics." + +### Value Semantics + +For a type with value semantics, variable initialization, assignment, +and argument-passing (hereafter, “the big three operations”) each create +an *independently modifiable copy* of the source value that is +*interchangeable with the source*. [^1] + +If `T` has value semantics, the `f`s below are all equivalent: + + func f1() -> T { + var x : T + return x + } + + func f2() -> T { + var x : T + var y = x + return y // a copy of x is equivalent to x + } + + func f2a() -> T { + var x : T + var y : T + y = x + return y // a copy of x is equivalent to x + } + + func f3() -> T { + var x : T + var y = x + y.mutate() // a copy of x is modifiable + return x // without affecting x + } + + func f3a() -> T { + var x : T + var y : T + y = x; + y.mutate() // a copy of x is modifiable + return x // without affecting x + } + + func g(x : T) { x.mutate() } + + func f4() -> T { + var x : T + g(x) // when x is passed by-value the copy + return x // is modifiable without affecting x + } + +### Reference Semantics + +Values of a type with reference semantics are only accessible +indirectly, via a reference. Although swift tries to hide this fact for +simplicity, for the purpose of this discussion it is important to note +that there are always *two* values in play: the value of the reference +itself and that of the object being referred to (a.k.a. the target). + +The programmer thinks of the target's value as primary, but it is never +used as a variable initializer, assigned, or passed as a function +argument. Conversely, the reference itself has full value semantics, but +the user never sees or names its type. The programmer expects that +copies of a reference share a target object, so modifications made via +one copy are visible as side-effects through the others. + +If `T` has reference semantics, the `f`s below are all equivalent: + + func f1(T x) { + x.mutate() + return x + } + + func f2(T x) -> T { + var y = x + y.mutate() // mutation through a copy of x + return x // is visible through x + } + + func f2a(T x) -> T { + var y : T + y = x + y.mutate() // mutation through a copy of x + return x // is visible through x + } + + func g(x : T) { x.mutate() } + + func f3(T x) -> T { + g(x) // when x is passed to a function, mutation + return x // through the parameter is visible through x + } + +### The Role of Mutation + +It's worth noting that in the absence of mutation, value semantics and +reference semantics are indistinguishable. You can easily prove that to +yourself by striking the calls to `mutate()` in each of the previous +examples, and seeing that the equivalences hold for any type. In fact, +the fundamental difference between reference and value semantics is that +**value semantics never creates multiple paths to the same mutable +state**. [^2] + +> **`struct` vs `class`** +> +> Although `struct`s were designed to support value semantics and +> `class`es were designed to support reference semantics, it would be +> wrong to assume that they are always used that way. As noted earlier, +> in the absence of mutation, value semantics and reference semantics +> are indistinguishable. Therefore, any immutable `class` trivially has +> value semantics (*and* reference semantics). +> +> Second, it's easy to implement a `struct` with reference semantics: +> simply keep the primary value in a `class` and refer to it through an +> instance variable. So, one cannot assume that a `struct` type has +> value semantics. `Array` could be seen (depending on how you view its +> value) as an example of a reference-semantics `struct` from the +> standard library. + +The Problem With Generics +------------------------- + +The classic Liskov principle says the semantics of operations on +`Duck`'s subtypes need to be consistent with those on `Duck` itself, so +that functions operating on `Duck`s still “work” when passed a +`Mallard`. More generally, for a function to make meaningful guarantees, +the semantics of its sub-operations need to be consistent regardless of +the actual argument types passed. + +The type of an argument passed by-value to an ordinary function is fully +constrained, so the “big three” have knowable semantics. The type of an +ordinary argument passed by-reference is constrained by subtype +polymorphism, where a (usually implicit) contract between base- and +sub-types can dictate consistency. + +However, the situation is different for functions with arguments of +protocol or parameterized type. In the absence of specific constraints +to the contrary, the semantics of the big three can vary. + +### Example + +For example, there's an algorithm called `cycle_length` that measures +the length of a cycle of states (e.g. the states of a pseudo-random +number generator). It needs to make one copy and do in-place mutation of +the state, rather than wholesale value replacement via assignment, which +might be expensive. + +Here’s a version of cycle\_length that works when state is a mutable +value type: + + func cycle_length( + s : State, mutate : ( [inout] State )->() + ) -> Int + requires State : EqualityComparable + { + State x = s // one copy // 1 + mutate(&x) // in-place mutation + Int n = 1 + while x != s { // 2 + mutate(&x) // in-place mutation + ++n + } + return n + } + +The reason the above breaks when the state is in a class instance is +that the intended copy in line 1 instead creates a new reference to the +same state, and the comparison in line 2 (regardless of whether we +decide `!=` does “identity” or “value” comparison) always succeeds. + +You can write a different implementation that only works on clonable +classes: + +You could also redefine the interface so that it works on both values +and clonable classes: + +However, this implementation makes O(N) separate copies of the state. I +don't believe there's a reasonable way write this so it works on +clonable classes, non-classes, and avoids the O(N) copies. [^3] + +### Class Identities are Values + +It's important to note that the first implementation of `cycle_length` +works when the state is the *identity*, rather than the *contents* of a +class instance. For example, imagine a circular linked list: + + class Node { + constructor(Int) { next = this; prev = this } + + // link two circular lists into one big cycle. + func join(otherNode : Node) -> () { ... } + + var next : WeakRef // identity of next node + var prev : WeakRef // identity of previous node + } + +We can measure the length of a cycle in these nodes as follows: + + cycle_length( someNode, (x: [inout] Node){ x = x.next } ) + +This is why so many generic algorithms seem to work on both `class`es +and non-`class`es: `class` *identities* work just fine as values. + +The Role of Moves +----------------- + +Further complicating matters is the fact that the big three operations +can be—and often are—combined in ways that mask the value/reference +distinction. In fact both of the following must be present in order to +observe a difference in behavior: + +1. Use of (one of) the big three operations on an object `x`, creating + shared mutable state iff `x` is a reference +2. In-place mutation of `x` *while a (reference) copy is extant* and + thus can be observed through the copy iff `x` is a reference. + +Take, for example, `swap`, which uses variable initialization and +assignment to exchange two values: + + func swap(lhs : [inout] T, rhs : [inout] T) + { + var tmp = lhs // big 3: initialization - ref copy in tmp + lhs = rhs // big 3: assignment - ref copy in lhs + rhs = tmp // big 3: assignment - no ref copies remain + } + +Whether `T` is a reference type makes no observable difference in the +behavior of `swap`. Why? Because although `swap` makes reference copies +to mutable state, the existence of those copies is encapsulated within +the algorithm, and it makes no in-place mutations. + +Any such algorithm can be implemented such that copy operations are +replaced by destructive *moves*, where the source value is not +(necessarily) preserved. Because movability is a weaker requirement than +copyability, it's reasonable to say that `swap` is built on *moves*, +rather than copies, in the same way that C++'s `std::find` is built on +input iterators rather than on forward iterators. + +We could imagine a hypothetical syntax for moving in swift, where +(unlike assignment) the value of the right-hand-side of the `<-` is not +necessarily preserved: + + var tmp <- lhs + lhs <- rhs + rhs <- tmp + +Such operations are safe to use in generic code without regard to the +differences between value- and reference- semantics. If this syntax were +extended to handle function arguments, it would cover the "big three" +operations: + + f(<-x) + +How to Build an Interesting Type with Value Semantics +----------------------------------------------------- + +Suppose we want to build a variable-sized data structure `X` with +(mutable) value semantics? How do we do it? + +If we make `` X` a ``class`, we automatically get reference semantics, so its value must be copied before each mutation, which is tedious and error-prone. Its public mutating interface must be in terms of free functions (not methods), so that the original reference value can be passed`\[inout\]`and overwritten. Since there's no user access to the reference count, we can't determine that we hold the only reference to the value, so we can't optimize copy-on-write, even in single-threaded programs. In multi-threaded programs, where each mutation implies synchronization on the reference count, the costs are even higher. If we make the type a`struct`, you have only two ways to create variable-sized data: 1. Hold a type with reference semantics as an instance variable. Unfortunately, this is really nothing new; we must still implement copy-on-write. We can, however, use methods for mutation in lieu of free functions. 2. Use discriminated unions (`union`). Interestingly, a datatype built with`union\`\` automatically has value semantics. However, + +: there vocabulary of efficient data structures that can be built this + way is extremely limited. For example, while a singly-linked list is + trivial to implement, an efficient doubly-linked list is + effectively impossible. + +------------------------------------------------------------------------ + +[^1]: Technically, copies of objects with value semantics are + interchangeable until they're mutated. Thereafter, the copies are + interchangeable except insofar as it matters what value type they + are *aggregated into*. + +[^2]: Note that this definition *does* allow for value semantics using + copy-on-write + +[^3]: I can think of a language extension that would allow this, but it + requires creating a protocol for generic copying, adding compiler + magic to get both classes and structs to conform to it, and telling + generic algorithm and container authors to use that protocol instead + of `=`, which IMO is really ugly and probably not worth the cost. diff --git a/docs/proposals/ValueSemantics.rst b/docs/proposals/ValueSemantics.rst deleted file mode 100644 index 3541d407e0a09..0000000000000 --- a/docs/proposals/ValueSemantics.rst +++ /dev/null @@ -1,380 +0,0 @@ -:orphan: - -.. @raise litre.TestsAreMissing -.. _ValueSemantics: - -========================== - Value Semantics in Swift -========================== - -:Abstract: Swift is the first language to take Generic Programming - seriously that also has both value and reference types. The - (sometimes subtle) differences between the behaviors of value and - reference types create unique challenges for generic programs that we - have not yet addressed. This paper surveys related problems - and explores some possible solutions. - - -Definitions -=========== - -I propose the following definitions of "value semantics" and -"reference semantics." - -Value Semantics ---------------- - -For a type with value semantics, variable initialization, assignment, -and argument-passing (hereafter, “the big three operations”) each -create an *independently modifiable copy* of the source value that is -*interchangeable with the source*. [#interchange]_ - -If ``T`` has value semantics, the ``f``\ s below are all equivalent:: - - func f1() -> T { - var x : T - return x - } - - func f2() -> T { - var x : T - var y = x - return y // a copy of x is equivalent to x - } - - func f2a() -> T { - var x : T - var y : T - y = x - return y // a copy of x is equivalent to x - } - - func f3() -> T { - var x : T - var y = x - y.mutate() // a copy of x is modifiable - return x // without affecting x - } - - func f3a() -> T { - var x : T - var y : T - y = x; - y.mutate() // a copy of x is modifiable - return x // without affecting x - } - - func g(x : T) { x.mutate() } - - func f4() -> T { - var x : T - g(x) // when x is passed by-value the copy - return x // is modifiable without affecting x - } - - -Reference Semantics -------------------- - -Values of a type with reference semantics are only accessible -indirectly, via a reference. Although swift tries to hide this fact -for simplicity, for the purpose of this discussion it is important to -note that there are always *two* values in play: the value of the -reference itself and that of the object being referred to (a.k.a. the -target). - -The programmer thinks of the target's value as primary, but it is -never used as a variable initializer, assigned, or passed as a -function argument. Conversely, the reference itself has full value -semantics, but the user never sees or names its type. The programmer -expects that copies of a reference share a target object, so -modifications made via one copy are visible as side-effects through -the others. - -If ``T`` has reference semantics, the ``f``\ s below are all -equivalent:: - - func f1(T x) { - x.mutate() - return x - } - - func f2(T x) -> T { - var y = x - y.mutate() // mutation through a copy of x - return x // is visible through x - } - - func f2a(T x) -> T { - var y : T - y = x - y.mutate() // mutation through a copy of x - return x // is visible through x - } - - func g(x : T) { x.mutate() } - - func f3(T x) -> T { - g(x) // when x is passed to a function, mutation - return x // through the parameter is visible through x - } - -The Role of Mutation --------------------- - -It's worth noting that in the absence of mutation, value semantics and -reference semantics are indistinguishable. You can easily prove that -to yourself by striking the calls to ``mutate()`` in each of the -previous examples, and seeing that the equivalences hold for any type. -In fact, the fundamental difference between reference and value -semantics is that **value semantics never creates multiple paths to -the same mutable state**. [#cow]_ - -.. Admonition:: ``struct`` vs ``class`` - - Although ``struct``\ s were designed to support value semantics and - ``class``\ es were designed to support reference semantics, it would - be wrong to assume that they are always used that way. As noted - earlier, in the absence of mutation, value semantics and reference - semantics are indistinguishable. Therefore, any immutable ``class`` - trivially has value semantics (*and* reference semantics). - - Second, it's easy to implement a ``struct`` with reference semantics: - simply keep the primary value in a ``class`` and refer to it through - an instance variable. So, one cannot assume that a ``struct`` type - has value semantics. ``Array`` could be seen (depending on how you - view its value) as an example of a reference-semantics ``struct`` - from the standard library. - -The Problem With Generics -========================= - -The classic Liskov principle says the semantics of operations on -``Duck``\ 's subtypes need to be consistent with those on ``Duck`` itself, -so that functions operating on ``Duck``\ s still “work” when passed a -``Mallard``. More generally, for a function to make meaningful -guarantees, the semantics of its sub-operations need to be consistent -regardless of the actual argument types passed. - -The type of an argument passed by-value to an ordinary function is -fully constrained, so the “big three” have knowable semantics. The -type of an ordinary argument passed by-reference is constrained by -subtype polymorphism, where a (usually implicit) contract between -base- and sub-types can dictate consistency. - -However, the situation is different for functions with arguments of -protocol or parameterized type. In the absence of specific -constraints to the contrary, the semantics of the big three can vary. - -Example -------- - -For example, there's an algorithm called ``cycle_length`` that -measures the length of a cycle of states (e.g. the states of a -pseudo-random number generator). It needs to make one copy and do -in-place mutation of the state, rather than wholesale value -replacement via assignment, which might be expensive. - -Here’s a version of cycle_length that works when state is a mutable -value type:: - - func cycle_length( - s : State, mutate : ( [inout] State )->() - ) -> Int - requires State : EqualityComparable - { - State x = s // one copy // 1 - mutate(&x) // in-place mutation - Int n = 1 - while x != s { // 2 - mutate(&x) // in-place mutation - ++n - } - return n - } - -The reason the above breaks when the state is in a class instance is -that the intended copy in line 1 instead creates a new reference to -the same state, and the comparison in line 2 (regardless of whether we -decide ``!=`` does “identity” or “value” comparison) always succeeds. - -You can write a different implementation that only works on clonable -classes: - -.. parsed-literal:: - - // Various random number generators will implement this interface - abstract class RandomNumberGenerator - : Clonable, Equalable - { - func nextValue() -> Int - } - - func cycle_length( - s : State, mutate : ( [inout] State )->() - ) -> Int - requires State : EqualityComparable, **Clonable** - { - State x = s\ **.clone()** - Int n = 1 - while **! x.equal(s)** { - *etc.* - } - - RandomNumberGenerator x = new MersenneTwister() - print( - cycle_length(x, (x : [inout] RandomNumberGenerator) { x.nextValue() }) - ) - -You could also redefine the interface so that it works on both values and -clonable classes: - -.. parsed-literal:: - - func cycle_length( - s : State, - **next : (x : State)->State,** - **equal : ([inout] x : State, [inout] y : State)->Bool** - ) -> Int - requires State : EqualityComparable - { - State **x = next(s)** - Int n = 1 - while **!equal(x, s)** { - **x = next(x)** - ++n - } - return n - } - -However, this implementation makes O(N) separate copies of the state. -I don't believe there's a reasonable way write this so it works on -clonable classes, non-classes, and avoids the O(N) -copies. [#extension]_ - -Class Identities are Values ---------------------------- - -It's important to note that the first implementation of -``cycle_length`` works when the state is the *identity*, rather than -the *contents* of a class instance. For example, imagine a circular -linked list:: - - class Node { - constructor(Int) { next = this; prev = this } - - // link two circular lists into one big cycle. - func join(otherNode : Node) -> () { ... } - - var next : WeakRef // identity of next node - var prev : WeakRef // identity of previous node - } - -We can measure the length of a cycle in these nodes as follows:: - - cycle_length( someNode, (x: [inout] Node){ x = x.next } ) - -This is why so many generic algorithms seem to work on both -``class``\ es and non-``class``\ es: ``class`` *identities* -work just fine as values. - -The Role of Moves -================= - -Further complicating matters is the fact that the big three operations -can be—and often are—combined in ways that mask the value/reference -distinction. In fact both of the following must be present in order -to observe a difference in behavior: - -1. Use of (one of) the big three operations on an object ``x``, - creating shared mutable state iff ``x`` is a reference - -2. In-place mutation of ``x`` *while a (reference) copy is extant* and - thus can be observed through the copy iff ``x`` is a reference. - -Take, for example, ``swap``, which uses variable initialization and -assignment to exchange two values:: - - func swap(lhs : [inout] T, rhs : [inout] T) - { - var tmp = lhs // big 3: initialization - ref copy in tmp - lhs = rhs // big 3: assignment - ref copy in lhs - rhs = tmp // big 3: assignment - no ref copies remain - } - -Whether ``T`` is a reference type makes no observable difference in -the behavior of ``swap``. Why? Because although ``swap`` makes -reference copies to mutable state, the existence of those copies is -encapsulated within the algorithm, and it makes no in-place mutations. - -Any such algorithm can be implemented such that copy operations are -replaced by destructive *moves*, where the source value is not -(necessarily) preserved. Because movability is a weaker requirement -than copyability, it's reasonable to say that ``swap`` is built on -*moves*, rather than copies, in the same way that C++'s ``std::find`` -is built on input iterators rather than on forward iterators. - -We could imagine a hypothetical syntax for moving in swift, where -(unlike assignment) the value of the right-hand-side of the ``<-`` is -not necessarily preserved:: - - var tmp <- lhs - lhs <- rhs - rhs <- tmp - -Such operations are safe to use in generic code without regard to the -differences between value- and reference- semantics. If this syntax -were extended to handle function arguments, it would cover the "big -three" operations:: - - f(<-x) - -How to Build an Interesting Type with Value Semantics -===================================================== - -Suppose we want to build a variable-sized data structure ``X`` with -(mutable) value semantics? How do we do it? - -If we make ``X` a ``class``, we automatically get reference semantics, so -its value must be copied before each mutation, which is tedious and -error-prone. Its public mutating interface must be in terms of free -functions (not methods), so that the original reference value can be -passed ``[inout]`` and overwritten. Since there's no user access to the -reference count, we can't determine that we hold the only reference to -the value, so we can't optimize copy-on-write, even in single-threaded -programs. In multi-threaded programs, where each mutation implies -synchronization on the reference count, the costs are even higher. - -If we make the type a ``struct``, you have only two ways to create -variable-sized data: - -1. Hold a type with reference semantics as an instance variable. - Unfortunately, this is really nothing new; we must still implement - copy-on-write. We can, however, use methods for mutation in lieu - of free functions. - -2. Use discriminated unions (``union``). Interestingly, a datatype - built with ``union`` automatically has value semantics. However, - there vocabulary of efficient data structures that can be built - this way is extremely limited. For example, while a singly-linked - list is trivial to implement, an efficient doubly-linked list is - effectively impossible. - ----- - -.. [#interchange] Technically, copies of objects with value semantics - are interchangeable until they're mutated. - Thereafter, the copies are interchangeable except - insofar as it matters what value type they are - *aggregated into*. - -.. [#cow] Note that this definition *does* allow for value semantics - using copy-on-write - -.. [#extension] I can think of a language extension that would allow - this, but it requires creating a protocol for generic - copying, adding compiler magic to get both classes and - structs to conform to it, and telling generic - algorithm and container authors to use that protocol - instead of ``=``, which IMO is really ugly and - probably not worth the cost. diff --git a/docs/proposals/WholeModuleOptimization.md b/docs/proposals/WholeModuleOptimization.md new file mode 100644 index 0000000000000..d22eaae92efd8 --- /dev/null +++ b/docs/proposals/WholeModuleOptimization.md @@ -0,0 +1,193 @@ +Whole Module Optimization in Swift +================================== + +State of Whole Module Optimization in Swift 2.0 +----------------------------------------------- + +Since Swift 1.2 / Xcode 6.3, the Swift optimizer has included support +for whole module optimization (WMO). + +To date (Swift 2.0 / Xcode 7), the differences in the optimization +pipeline and specific optimization passes when WMO is enabled have been +relatively minimal, and have provided high value at low implementation +cost. Examples of this include inferring final on internal methods, and +removing functions that are not referenced within the module and cannot +be referenced from outside the module. + +Additionally, compiling with WMO has some natural consequences that +require no enhancements to passes. For example the increased scope of +compilation that results from having the entire module available makes +it possible for the inliner to inline functions that it would otherwise +not be able to inline in normal separate compilation. Other +optimizations similarly benefit, for example generic specialization +(since it has more opportunities for specializae) and function signature +optimization (since it has more call sites to rewrite). + +Whole Module Optimization for Swift.Next +---------------------------------------- + +As it stands, WMO provides significant benefit with minimal complexity, +but also has many areas in which it can be improved. Some of these areas +for improvement are architectural, e.g. improvements to the pass +management scheme, while others involve adding new interprocedural +analyses and updating existing passes to make use of the results of +these analyses. + +### Passes and Pass Management + +Our current pass management scheme intersperses module passes and +function passes in a way that results in module passes being run on +functions that are partially optimized, which is suboptimal. Consider +inlining as one example. If we run the inliner when callee functions are +only partially optimized we may make different inlining decisions in a +caller than if we ran it on a caller only after its callees have been +fully optimized. This particular issue and others mentioned below are +not specific to WMO per-se, but are more generally a problem for any +interprocedural optimization that we currently do (most of which happen +in per-file builds as well). + +This also affects interprocedural optimizations where we can compute +summary information for a function and use that information when +optimizing callers of that function. We can obtain better results by +processing strongly connected components (SCCs) of the call graph rather +than individual functions, and running the full optimization pipeline on +a given SCC before moving to the next SCC in a post-order traversal of +the call graph (i.e. bottom-up). A similar approach can be taken for +running transforms that benefit from information that is propagated in +reverse-post-order in the call graph (i.e. top-down). + +Processing one SCC at a time versus one function at a time is primarily +for the benefit of improved analyses. For example consider escape +analysis. If there is an SCC in the call graph and we process one +function at a time, there are cases where we would have to be +pessimistic and assume a value escapes, when in fact the value may be +used within the SCC such that it never escapes. The same pessimism can +happen in other analyses, e.g. dead argument analysis. + +In our current set of passes, several are implemented as +SILModuleTransforms but simply iterate over the functions in the module +in whatever order they happen to be in and do not appear to benefit from +being module passes at this time (although some would benefit from +optimizing the functions in bottom-up order in the call graph). Other +SILModuleTransforms currently walk the functions bottom-up in the call +graph, and do gain some benefit from doing so. + +Moving forward we should eliminate the notion of module passes and +instead have SCC passes as well as the existing function passes. We +should change the pass manager to run the SCC passes as a bottom-up +traversal of the call graph. As previously mentioned we may also want to +consider having the flexibility of being able to run passes in a +top-down order (so we could create a pass manager with passes that +benefit from running in this order, not because we would want to run any +arbitrary pass in that order). + +For each SCC we would run the entire set of passes before moving on to +the next SCC, so when we go to optimize an SCC any functions that it +calls (when going bottom-up - or calls into it when going top-down) have +been fully optimized. As part of this, analyses that benefit from +looking at an SCC-at-a-time will need to be modified to do so. Existing +analyses that would benefit from looking at an SCC-at-a-time but have +not yet been updated to do so can be run on each function in the SCC in +turn, producing potentially pessimistic results. In time these can be +updated to analyze an entire SCC. Likewise function passes would be run +on each function in the SCC in turn. SCC function passes would be handed +an entire SCC and be able to ask for the analysis results for that SCC, +but can process each function individually based on those results. + +**TBD: Do we really need SCC transforms at all, or is it sufficient to +simply have function transforms that are always passed an SCC, and have +them ask for the results of an analyses for the entire SCC and then +iterate over all functions in the SCC?** + +In some cases we have transforms that generate new work in a top-down +fashion, for example the devirtualizer as well as any pass that clones. +These can be handled by allowing function passes to push new work at the +top of the stack of work items, and then upon finishing a pass the pass +pipeline will be restarted with those new functions at the top of the +work stack, and the previous function buried beneath them, to be +reprocessed after all the callees are processed. + +**TBD: This may result in re-running some early passes multiple times +for any given function, and it may mean we want to front-load the passes +that generate new callees like this so that they are early in the +pipeline. We could also choose not to rerun portions of the pipeline on +the function that's already had some processing done on it.**\* + +### SCC-based analyses + +There are a variety of analyses that can be done on an SCC to produce +better information than can be produced by looking at a single function +at a time, even when processing in (reverse-)post-order on the call +graph. For example, a dead argument analysis can provide information +about arguments that are not actually used, making it possible for +optimizations like DCE and dead-store optimization (which we do as part +of load/store opts) to eliminate code, independent of a pass like +function signature optimization running (and thus we eliminate a +phase-ordering issue). It benefits from looking at an SCC-at-a-time +because some arguments getting passed into the SCC might be passed +around the SCC, but are never used in any other way by a function within +the SCC (and are never passed outside of the SCC). + +Similarly, escape analysis, alias analysis, and mod/ref analysis benefit +from analyzing an SCC rather than a function. + +This list is not meant to be exhaustive, but is probably a good initial +set of analyses to plan for once we have the new pass framework up. + +### Incremental Whole Module Optimization + +Building with WMO enabled can result in slower build times due to +reduced parallelization of the build process. We currently parallelize +some of the last-mile optimization and code generation through LLVM, but +do not attempt to parallelize any of the SIL passes (see the next +section). + +With that in mind, it seems very worthwhile to examine doing incremental +WMO in order to minimize the amount of work done for each compile. + +One way to accomplish this is to serialize function bodies after +canonical SIL is produced (i.e. after the mandatory passes) and only +reoptimize those functions which change between builds. Doing just this, +however, is not sufficient since we've used information gleaned from +examining other functions, and we've also done things like inline +functions into one another. As a result, we need to track dependencies +between functions in order to properly do incremental compilation during +WMO. + +Some approaches to tracking these dependencies could be very burdensome, +requiring passes to explicitly track exactly which information they +actually use during optimization. This seems error prone and difficult +to maintain. + +Another approach might be to recompile adjacent functions in the call +graph when a given function changes. This might be somewhat practical if +we only have analyses which propagate information bottom-up, but it +would be more expensive than necessary, and impractical if we also have +analyses that propagate information top-down since it could result in a +full recompile of the module in the worst case. + +A more reasonable approach would be to serialize the results of the +interprocedural analyses at the end of the pass pipeline, and use these +serialized results to drive some of the dependency tracking (along with +some manual tracking, e.g. tracking which functions are inlined at which +call sites). These serialized analysis results would then be compared +against the results of running the same analyses at the end of the +compilation pipeline on any function which has changed since the +previous compile. If the results of an analysis changes, the functions +which use the results of that analysis would also need to be recompiled. + +**TBD: Properly tracking dependencies for functions generated from other +functions via cloning. Is this any different from tracking for +inlining?** + +### Parallel Whole Module Optimization + +We could also explore the possibility of doing more work in parallel +during WMO builds. For example, it may be feasible to run the SIL +optimization passes in parallel. It may also be feasible to do IRGen in +parallel, although there are shared mutating structures that would need +to be guarded. + +It's TBD whether this is actually going to be practical and worthwhile, +but it seems worth investigating and scoping out the work involved to +some first-level of approximation. diff --git a/docs/proposals/WholeModuleOptimization.rst b/docs/proposals/WholeModuleOptimization.rst deleted file mode 100644 index f391272a216c6..0000000000000 --- a/docs/proposals/WholeModuleOptimization.rst +++ /dev/null @@ -1,206 +0,0 @@ -:orphan: - -================================== -Whole Module Optimization in Swift -================================== - -State of Whole Module Optimization in Swift 2.0 -=============================================== - -Since Swift 1.2 / Xcode 6.3, the Swift optimizer has included support -for whole module optimization (WMO). - -To date (Swift 2.0 / Xcode 7), the differences in the optimization -pipeline and specific optimization passes when WMO is enabled have -been relatively minimal, and have provided high value at low -implementation cost. Examples of this include inferring final on -internal methods, and removing functions that are not referenced -within the module and cannot be referenced from outside the module. - -Additionally, compiling with WMO has some natural consequences that -require no enhancements to passes. For example the increased scope of -compilation that results from having the entire module available makes -it possible for the inliner to inline functions that it would -otherwise not be able to inline in normal separate compilation. Other -optimizations similarly benefit, for example generic specialization -(since it has more opportunities for specializae) and function -signature optimization (since it has more call sites to rewrite). - - -Whole Module Optimization for Swift.Next -======================================== - -As it stands, WMO provides significant benefit with minimal complexity, -but also has many areas in which it can be improved. Some of these -areas for improvement are architectural, e.g. improvements to the pass -management scheme, while others involve adding new interprocedural -analyses and updating existing passes to make use of the results of -these analyses. - -Passes and Pass Management --------------------------- - -Our current pass management scheme intersperses module passes and -function passes in a way that results in module passes being run on -functions that are partially optimized, which is suboptimal. Consider -inlining as one example. If we run the inliner when callee functions -are only partially optimized we may make different inlining decisions -in a caller than if we ran it on a caller only after its callees have -been fully optimized. This particular issue and others mentioned below -are not specific to WMO per-se, but are more generally a problem for -any interprocedural optimization that we currently do (most of which -happen in per-file builds as well). - -This also affects interprocedural optimizations where we can compute -summary information for a function and use that information when -optimizing callers of that function. We can obtain better results by -processing strongly connected components (SCCs) of the call graph -rather than individual functions, and running the full optimization -pipeline on a given SCC before moving to the next SCC in a post-order -traversal of the call graph (i.e. bottom-up). A similar approach can -be taken for running transforms that benefit from information that is -propagated in reverse-post-order in the call graph (i.e. top-down). - -Processing one SCC at a time versus one function at a time is -primarily for the benefit of improved analyses. For example consider -escape analysis. If there is an SCC in the call graph and we process -one function at a time, there are cases where we would have to be -pessimistic and assume a value escapes, when in fact the value may be -used within the SCC such that it never escapes. The same pessimism can -happen in other analyses, e.g. dead argument analysis. - -In our current set of passes, several are implemented as -SILModuleTransforms but simply iterate over the functions in the -module in whatever order they happen to be in and do not appear to -benefit from being module passes at this time (although some would -benefit from optimizing the functions in bottom-up order in the call -graph). Other SILModuleTransforms currently walk the functions -bottom-up in the call graph, and do gain some benefit from doing so. - -Moving forward we should eliminate the notion of module passes and -instead have SCC passes as well as the existing function passes. We -should change the pass manager to run the SCC passes as a bottom-up -traversal of the call graph. As previously mentioned we may also want -to consider having the flexibility of being able to run passes in a -top-down order (so we could create a pass manager with passes that -benefit from running in this order, not because we would want to run -any arbitrary pass in that order). - -For each SCC we would run the entire set of passes before moving on to -the next SCC, so when we go to optimize an SCC any functions that it -calls (when going bottom-up - or calls into it when going top-down) -have been fully optimized. As part of this, analyses that benefit from -looking at an SCC-at-a-time will need to be modified to do -so. Existing analyses that would benefit from looking at an -SCC-at-a-time but have not yet been updated to do so can be run on -each function in the SCC in turn, producing potentially pessimistic -results. In time these can be updated to analyze an entire -SCC. Likewise function passes would be run on each function in the SCC -in turn. SCC function passes would be handed an entire SCC and be able -to ask for the analysis results for that SCC, but can process each -function individually based on those results. - -**TBD: Do we really need SCC transforms at all, or is it sufficient to -simply have function transforms that are always passed an SCC, and -have them ask for the results of an analyses for the entire SCC and -then iterate over all functions in the SCC?** - -In some cases we have transforms that generate new work in a top-down -fashion, for example the devirtualizer as well as any pass that -clones. These can be handled by allowing function passes to push new -work at the top of the stack of work items, and then upon finishing a -pass the pass pipeline will be restarted with those new functions at -the top of the work stack, and the previous function buried beneath -them, to be reprocessed after all the callees are processed. - -**TBD: This may result in re-running some early passes multiple times -for any given function, and it may mean we want to front-load the -passes that generate new callees like this so that they are early in -the pipeline. We could also choose not to rerun portions of the -pipeline on the function that's already had some processing done on -it.*** - -SCC-based analyses ------------------- - -There are a variety of analyses that can be done on an SCC to produce -better information than can be produced by looking at a single -function at a time, even when processing in (reverse-)post-order on -the call graph. For example, a dead argument analysis can provide -information about arguments that are not actually used, making it -possible for optimizations like DCE and dead-store optimization (which -we do as part of load/store opts) to eliminate code, independent of a -pass like function signature optimization running (and thus we -eliminate a phase-ordering issue). It benefits from looking at an -SCC-at-a-time because some arguments getting passed into the SCC might -be passed around the SCC, but are never used in any other way by a -function within the SCC (and are never passed outside of the SCC). - -Similarly, escape analysis, alias analysis, and mod/ref analysis -benefit from analyzing an SCC rather than a function. - -This list is not meant to be exhaustive, but is probably a good -initial set of analyses to plan for once we have the new pass -framework up. - -Incremental Whole Module Optimization -------------------------------------- - -Building with WMO enabled can result in slower build times due to -reduced parallelization of the build process. We currently parallelize -some of the last-mile optimization and code generation through LLVM, -but do not attempt to parallelize any of the SIL passes (see the next -section). - -With that in mind, it seems very worthwhile to examine doing -incremental WMO in order to minimize the amount of work done for each -compile. - -One way to accomplish this is to serialize function bodies after -canonical SIL is produced (i.e. after the mandatory passes) and only -reoptimize those functions which change between builds. Doing just -this, however, is not sufficient since we've used information gleaned -from examining other functions, and we've also done things like inline -functions into one another. As a result, we need to track dependencies -between functions in order to properly do incremental compilation -during WMO. - -Some approaches to tracking these dependencies could be very -burdensome, requiring passes to explicitly track exactly which -information they actually use during optimization. This seems error -prone and difficult to maintain. - -Another approach might be to recompile adjacent functions in the call -graph when a given function changes. This might be somewhat practical -if we only have analyses which propagate information bottom-up, but it -would be more expensive than necessary, and impractical if we also -have analyses that propagate information top-down since it could -result in a full recompile of the module in the worst case. - -A more reasonable approach would be to serialize the results of the -interprocedural analyses at the end of the pass pipeline, and use -these serialized results to drive some of the dependency tracking -(along with some manual tracking, e.g. tracking which functions are -inlined at which call sites). These serialized analysis results would -then be compared against the results of running the same analyses at -the end of the compilation pipeline on any function which has changed -since the previous compile. If the results of an analysis changes, the -functions which use the results of that analysis would also need to be -recompiled. - -**TBD: Properly tracking dependencies for functions generated from -other functions via cloning. Is this any different from tracking for -inlining?** - -Parallel Whole Module Optimization ----------------------------------- - -We could also explore the possibility of doing more work in parallel -during WMO builds. For example, it may be feasible to run the SIL -optimization passes in parallel. It may also be feasible to do IRGen -in parallel, although there are shared mutating structures that would -need to be guarded. - -It's TBD whether this is actually going to be practical and -worthwhile, but it seems worth investigating and scoping out the work -involved to some first-level of approximation. diff --git a/docs/proposals/archive/Memory and Concurrency Model.md b/docs/proposals/archive/Memory and Concurrency Model.md new file mode 100644 index 0000000000000..353843b9321ff --- /dev/null +++ b/docs/proposals/archive/Memory and Concurrency Model.md @@ -0,0 +1,372 @@ +Swift Memory and Concurrency Model +================================== + +> **warning** +> +> This is a very early design document discussing the features of + +> a possible Swift concurrency model. It should not be taken as a plan +> of record. + +The goal of this writeup is to provide a safe and efficient way to +model, design, and implement concurrent applications in Swift. It is +believed that it will completely eliminate data races and reduce +deadlocks in swift apps, and will allow for important performance wins +as well. This happens by eliminating shared mutable data, locks, and the +need for most atomic memory accesses. The model is quite different from +what traditional unix folks are used to though. :-) + +As usual, Swift will eventually support unsafe constructs, so if it +turns out that the model isn't strong enough for some particular use +case, that coder can always fall back to the hunting for concurrency +with arrows and bone knives instead of using the native language +support. Ideally this will never be required though. + +This starts by describing the basic model, then talks about issues and +some possible extensions to improve the model. + +Language Concepts +----------------- + +The **unit of concurrency** in Swift is an Actor. As an execution +context, an actor corresponds to a lightweight thread or dispatch queue +that can respond to a declared list of async messages. Each actor has +its own private set of mutable data, can communicate with other actors +by sending them async messages, and can lock each other to perform +synchronous actions. + +Only one message can be active at a time (you can also extend the model +to allow read/writer locks as a refinement) in an actor. Through the +type system and runtime, we make it impossible for an actor to read or +change another actor's mutable data: all mutable data is private to some +actor. When a message is sent from one actor to another, the argument is +deep copied to ensure that two actors can never share mutable data. + +To achieve this, there are three important kinds of memory object in +Swift. Given a static type, it is obvious what kind it is from its +definition. These kinds are: + +1. **Immutable Data** - Immutable data (which can have a constructor, + but whose value cannot be changed after it completes) is sharable + across actors, and it would make sense to unique them + where possible. Immutable data can (transitively) point to other + immutable data, but it isn't valid (and the compiler rejects) + immutable data that is pointing to mutable data. For example: + + struct [immutable,inout] IList { data : int, next : List } + ... + var a = IList(1, IList(2, IList(3, nil))) + ... + a.data = 42 // error, can't mutate field of immutable 'IList' type! + a.next = nil // error, same deal + a = IList(2, IList(3, nil)) // ok, "a" itself is mutable, it now points to something new. + a = IList(1, a) // ok + +> Strings are a very important example of (inout) immutable data. + +2. **Normal mutable data** - Data (objects, structs, etc) are mutable + by default, and are thus local to the containing actor. Mutable data + can point to immutable data with no problem, and since it is local + to an actor, no synchronization or atomic accesses are ever required + for any mutable data. For example: + + struct [inout] MEntry { x : int, y : int, list : IList } + ... + var b = MEntry(4, 2, IList(1, nil)) + b.x = 4 // ok + b.y = 71 // ok + b.list = nil // ok + b.list = IList(2, nil) // ok + b.list.data = 42 // error, can't change immutable data. + + As part of mutable data, it is worth pointing out that mutable + "global variables" in swift are not truly global, they are local to + the current actor (somewhat similar to "thread local storage", or + perhaps to "an ivar on the actor"). Immutable global variables (like + lookup tables) are simple immutable data just like today. Global + variables with "static constructors / initializers" in the C++ sense + have their constructor lazily run on the first use of the variable. + + Not having mutable shared state also prevents race conditions from + turning into safety violations. Here's a description of the go issue + which causes them to be unsafe when running multiple threads: + http://research.swtch.com/2010/02/off-to-races.html\_ + + Note that a highly desirable feature of the language is the ability + to take an instance of a mutable type and *make it immutable* + somehow, allowing it to be passed around by-value. It is unclear how + to achieve this, but perhaps typestate can magically make + it possible. + +3. **Actors** - In addition to being the unit of concurrency, Actors + have an address and are thus memory objects. Actors themselves can + have mutable fields, one actor can point to another, and mutable + data can point to an actor. However, any mutable data in an actor + can only be directly accessed by that actor, it isn't possible for + one actor to access another actor's mutable data. Also note that + actors and objects are completely unrelated: it is not possible for + an object to inherit from an actor, and it is not possible for an + actor to inherit from an object. Either can contain the other as an + ivar though. + + Syntax for actors is TBD and not super important for now, but here + is a silly multithreaded mandelbrot example, that does each pixel in + "parallel", to illustrate some ideas: + + func do_mandelbrot(x : float, y : float) -> int { + // details elided + } + + actor MandelbrotCalculator { + func compute(x : float, y : float, Driver D) { + var num_iters = do_mandelbrot(x, y) + D.collect_point(x, y, num_iters) + } + } + + actor Driver { + var result : image; // result and numpoints are mutable per-actor data. + var numpoints : int; + func main() { + result = new image() + foreach i in -2.0 ... 2.0 by 0.001 { + // Arbitrarily, create one MandelbrotCalculator for each row. + var MC = new MandelbrotCalculator() + foreach j in -2.0 ... 2.0 by 0.001 { + MC.compute(i, j, self) + ++numpoints; + } + } + } + + func collect_point(x : float, y : float, num_iters : int) { + result.setPoint(x, y, Color(num_iters, num_iters, num_iters)) + if (--numpoints == 0) + draw(result) + } + } + + Though actors have mutable data (like 'result' and 'numpoints'), + there is no need for any synchronization on that mutable data. + + One of the great things about this model (in my opinion) is that it + gives programmers a way to reason about granularity, and the data + copy/sharing issue gives them something very concrete and + understandable that they can use to make design decisions when + building their app. While it is a common pattern to have one class + that corresponds to a thread in C++ and ObjC, this is an informal + pattern -- baking this into the language with actors and giving a + semantic difference between objects and actors makes the tradeoffs + crisp and easy to understand and reason about. + +Communicating with Actors +------------------------- + +As the example above shows, the primary and preferred way to communicate +with actors is through one-way asynchronous messages. Asynchronous +message sensed are nice because they cannot block, deadlock, or have +other bad effects. However, they aren't great for two things: 1) +invoking multiple methods on an actor that need to be synchronized +together, and 2) getting a value back from the actor. + +Sending multiple messages asynchronously +---------------------------------------- + +With the basic approach above, you can only perform actions on actors +that are built into the actor. For example, if you had an actor with two +methods: + + actor MyActor { + func foo() {…} + func bar() {…} + func getvalue() -> double {… } + } + +Then there is no way to perform a composite operation that needs to +"atomically" perform foo() and bar() without any other operations +getting in between. If you had code like this: + + var a : MyActor = … + a.foo() + a.bar() + +Then the foo/bar methods are both sent asynchronously, and (while they +would be ordered with respect to each other) there is no guarantee that +some other method wouldn't be run in between them. To handle this, the +async block structure can be used to submit a sequence of code that is +atomically run in the actor's context, e.g.: + + var a : MyActor = … + async a { + a.foo() + a.bar() + } + +This conceptually submits a closure to run in the context of the actor. +If you look at it this way, an async message send is conceptually +equivalent to an async block. As such, the original example was +equivalent to: + + var a : MyActor = … + async a { a.foo() } + async a { a.bar() } + +which makes it pretty clear that the two sends are separate from each +other. We could optionally require all accesses to an actor to be in an +async block, which would make this behavior really clear at the cost of +coding clarity. + +It is worth pointing out that you can't asynchronously call a message +and get its return value back. However, if the return value is ignored, +a message send can be performed. For example, "a.getvalue()" would be +fine so long as the result is ignored or if the value is in an explicit +async block structure. + +From an implementation perspective, the code above corresponds directly +to GCD's dispatch\_asynch on a per-actor queue. + +Performing synchronous operations +--------------------------------- + +Asynchronous calls are nice and define away the possibility of deadlock, +but at some point you need to get a return value back and async +programming is very awkward. To handle this, a 'synch' block is used. +For example, the following is valid: + + var x : double + synch a { + x = a.getvalue(); + } + +but this is not: + + var x = a.getvalue(); + +A synch block statement is directly related to dispatch\_sync and +conceptually locks the specified actor's lock/queue and performs the +block within its context. + +Memory Ownership Model +---------------------- + +Within an actor there is a question of how ownership is handled. It's +not in the scope of this document to say what the "one true model" is, +but here are a couple of interesting observations: + +1. **Automated reference counting** would be much more efficient in + this model than in ObjC, because the compiler statically knows + whether something is mutable data or is shared. Mutable data (e.g. + normal objects) can be ref counted with non-atomic reference + counting, which is 20-30x faster than atomic adjustments. Actors are + shared, so they'd have to have atomic ref counts, but they should be + much much less common than the normal objects in the program. + Immutable data is shared (and thus needs atomic reference counts) + but there are optimizations that can be performed since the edges in + the pointer graph can never change and cycles aren't possible in + immutable data. +2. **Garbage collection** for mutable data becomes a lot more + attractive than in ObjC for four reasons: 1) all GC is local to an + actor, so you don't need to stop the world to do a collection. 2) + actors have natural local quiescent points: when they have finished + servicing a message, if their dispatch queue is empty, they go + to sleep. If nothing else in the CPU needs the thread, it would be a + natural time to collect. 3) GC would be fully precise in swift, + unlike in ObjC, no conservative stack scanning or other hacks + are needed. 4) If GC is used for mutable data, it would make sense + to still use reference counting for actors themselves and especially + for immutable data, meaning that you'd have *no* "whole process" GC. +3. Each actor can use a **different memory management policy**: it is + completely fine for one actor to be GC and one actor to be ARC, and + another to be manually malloc/freed (and thus unsafe) because actors + can't reach each other's pointers. However, realistically, we will + still have to pick "the right" model, because different actors can + share the same code (e.g. they can instantiate the same objects) and + the compiled code has to implement the model the actor wants. + +Issues with this Model +---------------------- + +There are two significant issues with this model: 1) the amount of data +copying may be excessive if you have lots of messages each passing lots +of mutable data that is deep copied, and 2) the awkward nature of async +programming for some (common) classes of programming. For example, the +"branch and rejoin" pattern in the example requires a counter to know +when everyone rejoined, and we really want a "parallel for loop". + +I'd advocate implementing the simple model first, but once it is there, +there are several extensions that can help with these two problems: + +**No copy is needed for some important cases:** If you can prove +(through the type system) that an object graph has a single (unique) +pointer to it, the pointer value can be sent in the message and nil'd +out in the sender. In this way you're "transferring" ownership of the +subgraph from one actor to the other. It's not fully clear how to do +this though. Another similar example: if we add some way for an actor to +self destruct along with a message send, then it is safe for an actor to +transfer any and all of its mutable state to another actor when it +destroys itself, avoiding a copy. + +**Getters for trivial immutable actor fields**: If an actor has an ivar +with an immutable type, then we can make all stores to it atomic, and +allow other actors to access the ivar. Silly example: + + actor Window { + var title : string; // string is an immutable by-ref type. + ... + } + + ... + var x = new Window; + print(x.title) // ok, all stores will be atomic, an (recursively) immutable data is valid in all actors, so this is fine to load. + ... + +**Parallel for loops** and other constructs that don't guarantee that +each "thread" has its own non-shared mutable memory are very important +and not covered by this model at all. For example, having multiple +threads execute on different slices of the same array would require +copying the array to temporary disjoint memory spaces to do operations, +then recopy it back into place. This data copying can be awkward and +reduce the benefits of parallelism to make it non- profitable. + +There are multiple different ways to tackle this. We can just throw it +back into the programmer's lap and tell them that the behavior is +undefined if they get a race condition. This is fine for some systems +levels stuff, but defeats the purpose of having a safe language and is +clearly not good enough for mobile code. + +Another (more aggressive) approach is to provide a parallel for loop, +and use it as a hint that each iteration can be executed in parallel. It +would then be up to the implementation to try to prove the safety of +this (e.g. using dependence analysis), and if provable everything is +good. If not provable, then the implementation would have to compile it +as serial code, or use something like an STM approach to guarantee that +the program is correct or the error is detected. There is much work in +academia that can be tapped for this sort of thing. One nice thing about +this approach is that you'd always get full parallel performance if you +"disable checking", which could be done in a production build or +something. + +Some blue sky kinds of random thoughts +-------------------------------------- + +**Distributed Programming** - Since deep copy is part of the language +and "deep copy" is so similar to "serialization", it would be easy to do +a simple implementation of something like "Distributed Objects". The +primary additional thing that is required is for messages sent to actors +to be able to fail, which is required anyway. The granularity issues +that come up are similar in these two domains. + +**Immutable Data w/Synch and Lazy Faulting** - Not a fully baked idea, +but if you're heavily using immutable data to avoid copies, a +"distributed objects" implementation would suffer because it would have +to deep copy all the immutable data that the receiver doesn't have, +defeating the optimization. One approach to handling this is to treat +this as a data synch problem, and have the client fault pieces of the +immutable data subgraph in on demand, instead of eagerly copying it. + +**OpenCL Integration** with this model could be really natural: the GPU +is an inherently async device to talk to. + +**UNIX processes**: Actors in a shared address space with no shared +mutable data are related to processes in a unix app that share by +communicating with mmap etc. diff --git a/docs/proposals/archive/Memory and Concurrency Model.rst b/docs/proposals/archive/Memory and Concurrency Model.rst deleted file mode 100644 index 82dc4bd50d986..0000000000000 --- a/docs/proposals/archive/Memory and Concurrency Model.rst +++ /dev/null @@ -1,371 +0,0 @@ -:orphan: - -.. @raise litre.TestsAreMissing -.. _MemoryAndConcurrencyModel: - -Swift Memory and Concurrency Model -================================== - -.. warning:: This is a very early design document discussing the features of - a possible Swift concurrency model. It should not be taken as a plan of - record. - -The goal of this writeup is to provide a safe and efficient way to model, -design, and implement concurrent applications in Swift. It is believed that it -will completely eliminate data races and reduce deadlocks in swift apps, and -will allow for important performance wins as well. This happens by eliminating -shared mutable data, locks, and the need for most atomic memory accesses. The -model is quite different from what traditional unix folks are used to -though. :-) - -As usual, Swift will eventually support unsafe constructs, so if it turns out -that the model isn't strong enough for some particular use case, that coder can -always fall back to the hunting for concurrency with arrows and bone knives -instead of using the native language support. Ideally this will never be -required though. - -This starts by describing the basic model, then talks about issues and some -possible extensions to improve the model. - -Language Concepts ------------------ - -The **unit of concurrency** in Swift is an Actor. As an execution context, an -actor corresponds to a lightweight thread or dispatch queue that can respond to -a declared list of async messages. Each actor has its own private set of mutable -data, can communicate with other actors by sending them async messages, and can -lock each other to perform synchronous actions. - -Only one message can be active at a time (you can also extend the model to allow -read/writer locks as a refinement) in an actor. Through the type system and -runtime, we make it impossible for an actor to read or change another actor's -mutable data: all mutable data is private to some actor. When a message is sent -from one actor to another, the argument is deep copied to ensure that two actors -can never share mutable data. - -To achieve this, there are three important kinds of memory object in -Swift. Given a static type, it is obvious what kind it is from its -definition. These kinds are: - -1. **Immutable Data** - Immutable data (which can have a constructor, but whose - value cannot be changed after it completes) is sharable across actors, and it - would make sense to unique them where possible. Immutable data can - (transitively) point to other immutable data, but it isn't valid (and the - compiler rejects) immutable data that is pointing to mutable data. For - example:: - - struct [immutable,inout] IList { data : int, next : List } - ... - var a = IList(1, IList(2, IList(3, nil))) - ... - a.data = 42 // error, can't mutate field of immutable 'IList' type! - a.next = nil // error, same deal - a = IList(2, IList(3, nil)) // ok, "a" itself is mutable, it now points to something new. - a = IList(1, a) // ok - - Strings are a very important example of (inout) immutable data. - - - - -2. **Normal mutable data** - Data (objects, structs, etc) are mutable by - default, and are thus local to the containing actor. Mutable data can point - to immutable data with no problem, and since it is local to an actor, no - synchronization or atomic accesses are ever required for any mutable - data. For example:: - - struct [inout] MEntry { x : int, y : int, list : IList } - ... - var b = MEntry(4, 2, IList(1, nil)) - b.x = 4 // ok - b.y = 71 // ok - b.list = nil // ok - b.list = IList(2, nil) // ok - b.list.data = 42 // error, can't change immutable data. - - As part of mutable data, it is worth pointing out that mutable "global - variables" in swift are not truly global, they are local to the current actor - (somewhat similar to "thread local storage", or perhaps to "an ivar on the - actor"). Immutable global variables (like lookup tables) are simple immutable - data just like today. Global variables with "static constructors / - initializers" in the C++ sense have their constructor lazily run on the first - use of the variable. - - Not having mutable shared state also prevents race conditions from turning - into safety violations. Here's a description of the go issue which causes - them to be unsafe when running multiple threads: - `http://research.swtch.com/2010/02/off-to-races.html`_ - - Note that a highly desirable feature of the language is the ability to take - an instance of a mutable type and *make it immutable* somehow, allowing it to - be passed around by-value. It is unclear how to achieve this, but perhaps - typestate can magically make it possible. - -3. **Actors** - In addition to being the unit of concurrency, Actors have an - address and are thus memory objects. Actors themselves can have mutable - fields, one actor can point to another, and mutable data can point to an - actor. However, any mutable data in an actor can only be directly accessed by - that actor, it isn't possible for one actor to access another actor's mutable - data. Also note that actors and objects are completely unrelated: it is not - possible for an object to inherit from an actor, and it is not possible for - an actor to inherit from an object. Either can contain the other as an ivar - though. - - Syntax for actors is TBD and not super important for now, but here is a silly - multithreaded mandelbrot example, that does each pixel in "parallel", to - illustrate some ideas:: - - func do_mandelbrot(x : float, y : float) -> int { - // details elided - } - - actor MandelbrotCalculator { - func compute(x : float, y : float, Driver D) { - var num_iters = do_mandelbrot(x, y) - D.collect_point(x, y, num_iters) - } - } - - actor Driver { - var result : image; // result and numpoints are mutable per-actor data. - var numpoints : int; - func main() { - result = new image() - foreach i in -2.0 ... 2.0 by 0.001 { - // Arbitrarily, create one MandelbrotCalculator for each row. - var MC = new MandelbrotCalculator() - foreach j in -2.0 ... 2.0 by 0.001 { - MC.compute(i, j, self) - ++numpoints; - } - } - } - - func collect_point(x : float, y : float, num_iters : int) { - result.setPoint(x, y, Color(num_iters, num_iters, num_iters)) - if (--numpoints == 0) - draw(result) - } - } - - Though actors have mutable data (like 'result' and 'numpoints'), there is no - need for any synchronization on that mutable data. - - One of the great things about this model (in my opinion) is that it gives - programmers a way to reason about granularity, and the data copy/sharing - issue gives them something very concrete and understandable that they can use - to make design decisions when building their app. While it is a common - pattern to have one class that corresponds to a thread in C++ and ObjC, this - is an informal pattern -- baking this into the language with actors and - giving a semantic difference between objects and actors makes the tradeoffs - crisp and easy to understand and reason about. - -Communicating with Actors -------------------------- - -As the example above shows, the primary and preferred way to communicate with -actors is through one-way asynchronous messages. Asynchronous message sensed -are nice because they cannot block, deadlock, or have other bad -effects. However, they aren't great for two things: 1) invoking multiple methods -on an actor that need to be synchronized together, and 2) getting a value back -from the actor. - -Sending multiple messages asynchronously ----------------------------------------- - -With the basic approach above, you can only perform actions on actors that are -built into the actor. For example, if you had an actor with two methods:: - - actor MyActor { - func foo() {…} - func bar() {…} - func getvalue() -> double {… } - } - -Then there is no way to perform a composite operation that needs to "atomically" -perform foo() and bar() without any other operations getting in between. If you -had code like this:: - - var a : MyActor = … - a.foo() - a.bar() - -Then the foo/bar methods are both sent asynchronously, and (while they would be -ordered with respect to each other) there is no guarantee that some other method -wouldn't be run in between them. To handle this, the async block structure can -be used to submit a sequence of code that is atomically run in the actor's -context, e.g.:: - - var a : MyActor = … - async a { - a.foo() - a.bar() - } - -This conceptually submits a closure to run in the context of the actor. If you -look at it this way, an async message send is conceptually equivalent to an -async block. As such, the original example was equivalent to:: - - var a : MyActor = … - async a { a.foo() } - async a { a.bar() } - -which makes it pretty clear that the two sends are separate from each other. We -could optionally require all accesses to an actor to be in an async block, which -would make this behavior really clear at the cost of coding clarity. - -It is worth pointing out that you can't asynchronously call a message and get -its return value back. However, if the return value is ignored, a message send -can be performed. For example, "a.getvalue()" would be fine so long as the -result is ignored or if the value is in an explicit async block structure. - -From an implementation perspective, the code above corresponds directly to GCD's -dispatch_asynch on a per-actor queue. - -Performing synchronous operations ---------------------------------- - -Asynchronous calls are nice and define away the possibility of deadlock, but at -some point you need to get a return value back and async programming is very -awkward. To handle this, a 'synch' block is used. For example, the following is -valid:: - - var x : double - synch a { - x = a.getvalue(); - } - -but this is not:: - - var x = a.getvalue(); - -A synch block statement is directly related to dispatch_sync and conceptually -locks the specified actor's lock/queue and performs the block within its -context. - -Memory Ownership Model ----------------------- - -Within an actor there is a question of how ownership is handled. It's not in the -scope of this document to say what the "one true model" is, but here are a -couple of interesting observations: - -1. **Automated reference counting** would be much more efficient in this model - than in ObjC, because the compiler statically knows whether something is - mutable data or is shared. Mutable data (e.g. normal objects) can be ref - counted with non-atomic reference counting, which is 20-30x faster than - atomic adjustments. Actors are shared, so they'd have to have atomic ref - counts, but they should be much much less common than the normal objects in - the program. Immutable data is shared (and thus needs atomic reference - counts) but there are optimizations that can be performed since the edges in - the pointer graph can never change and cycles aren't possible in immutable - data. - -2. **Garbage collection** for mutable data becomes a lot more attractive than in - ObjC for four reasons: 1) all GC is local to an actor, so you don't need to - stop the world to do a collection. 2) actors have natural local quiescent - points: when they have finished servicing a message, if their dispatch queue - is empty, they go to sleep. If nothing else in the CPU needs the thread, it - would be a natural time to collect. 3) GC would be fully precise in swift, - unlike in ObjC, no conservative stack scanning or other hacks are needed. 4) - If GC is used for mutable data, it would make sense to still use reference - counting for actors themselves and especially for immutable data, meaning - that you'd have *no* "whole process" GC. - -3. Each actor can use a **different memory management policy**: it is completely - fine for one actor to be GC and one actor to be ARC, and another to be - manually malloc/freed (and thus unsafe) because actors can't reach each - other's pointers. However, realistically, we will still have to pick "the - right" model, because different actors can share the same code (e.g. they can - instantiate the same objects) and the compiled code has to implement the - model the actor wants. - -Issues with this Model ----------------------- - -There are two significant issues with this model: 1) the amount of data copying -may be excessive if you have lots of messages each passing lots of mutable data -that is deep copied, and 2) the awkward nature of async programming for some -(common) classes of programming. For example, the "branch and rejoin" pattern -in the example requires a counter to know when everyone rejoined, and we really -want a "parallel for loop". - -I'd advocate implementing the simple model first, but once it is there, there -are several extensions that can help with these two problems: - -**No copy is needed for some important cases:** If you can prove (through the -type system) that an object graph has a single (unique) pointer to it, the -pointer value can be sent in the message and nil'd out in the sender. In this -way you're "transferring" ownership of the subgraph from one actor to the -other. It's not fully clear how to do this though. Another similar example: if -we add some way for an actor to self destruct along with a message send, then it -is safe for an actor to transfer any and all of its mutable state to another -actor when it destroys itself, avoiding a copy. - -**Getters for trivial immutable actor fields**: If an actor has an ivar with an -immutable type, then we can make all stores to it atomic, and allow other actors -to access the ivar. Silly example:: - - actor Window { - var title : string; // string is an immutable by-ref type. - ... - } - - ... - var x = new Window; - print(x.title) // ok, all stores will be atomic, an (recursively) immutable data is valid in all actors, so this is fine to load. - ... - -**Parallel for loops** and other constructs that don't guarantee that each -"thread" has its own non-shared mutable memory are very important and not -covered by this model at all. For example, having multiple threads execute on -different slices of the same array would require copying the array to temporary -disjoint memory spaces to do operations, then recopy it back into place. This -data copying can be awkward and reduce the benefits of parallelism to make it -non- profitable. - -There are multiple different ways to tackle this. We can just throw it back into -the programmer's lap and tell them that the behavior is undefined if they get a -race condition. This is fine for some systems levels stuff, but defeats the -purpose of having a safe language and is clearly not good enough for mobile -code. - -Another (more aggressive) approach is to provide a parallel for loop, and use it -as a hint that each iteration can be executed in parallel. It would then be up -to the implementation to try to prove the safety of this (e.g. using dependence -analysis), and if provable everything is good. If not provable, then the -implementation would have to compile it as serial code, or use something like an -STM approach to guarantee that the program is correct or the error is -detected. There is much work in academia that can be tapped for this sort of -thing. One nice thing about this approach is that you'd always get full -parallel performance if you "disable checking", which could be done in a -production build or something. - -Some blue sky kinds of random thoughts --------------------------------------- - -**Distributed Programming** - Since deep copy is part of the language and "deep -copy" is so similar to "serialization", it would be easy to do a simple -implementation of something like "Distributed Objects". The primary additional -thing that is required is for messages sent to actors to be able to fail, which -is required anyway. The granularity issues that come up are similar in these two -domains. - -**Immutable Data w/Synch and Lazy Faulting** - Not a fully baked idea, but if -you're heavily using immutable data to avoid copies, a "distributed objects" -implementation would suffer because it would have to deep copy all the immutable -data that the receiver doesn't have, defeating the optimization. One approach to -handling this is to treat this as a data synch problem, and have the client -fault pieces of the immutable data subgraph in on demand, instead of eagerly -copying it. - -**OpenCL Integration** with this model could be really natural: the GPU is an -inherently async device to talk to. - -**UNIX processes**: Actors in a shared address space with no shared mutable data -are related to processes in a unix app that share by communicating with mmap -etc. - -.. _http://research.swtch.com/2010/02/off-to-races.html: http://research.swtch.com/2010/02/off-to-races.html - - diff --git a/docs/proposals/archive/ProgramStructureAndCompilationModel.md b/docs/proposals/archive/ProgramStructureAndCompilationModel.md new file mode 100644 index 0000000000000..a1f99908319b3 --- /dev/null +++ b/docs/proposals/archive/ProgramStructureAndCompilationModel.md @@ -0,0 +1,392 @@ +Swift Program Structure and Compilation Model +============================================= + +> **warning** +> +> This is a very early design document discussing the features of + +> a Swift build model and modules system. It should not be taken as a +> plan of record. + +Commentary +---------- + +The C spec only describes things up to translation unit granularity: no +discussion of file system layout, build system, linking, runtime +concepts of code (dynamic libraries, executables, plugins), dependence +between parts of a program, Versioning + SDKs, human factors like +management units, etc. It leaves all of this up to implementors to sort +out, and we got what the unix world defined in the 60's and 70's with +some minor stuff that could be shoehorned into the old unix toolchain +model without too much trouble. C also doesn't help with resources +(images etc), has a miserable incremental compilation model and many, +many, other issues. + +Swift should strive to make trivial programs really simple. Hello world +should just be something like: + + print("hello world") + +while also acknowledging and strongly supporting the real world demands +and requirements that library implementors (hey, that's us!) face every +day. In particular, note how the language elements (described below) +correspond directly to the business and management reality of the world: + +**Ownership Domain / Top Level Component**: corresponds to a product +that is shipped as a unit (Mac OS/X, iWork, Xcode), is a collection of +frameworks/dylibs and resources. Only acyclic dependences between +different domains is allowed. There is some correlation in concept here +to "umbrella headers" or "dyld shared cache" though it isn't exact. + +**Namespace**: Organizational structure within a domain, similar to C++ +or Java. Programmers can use or abuse them however they wish. + +**Subcomponent**: corresponds to an individual team or management unit, +is one dylib + optional resources. All contributing source files and +resources live in one directory (with optional subdirs), and have a +single "project file". Can contribute to multiple namespaces. The +division of a domain into components is an implementation detail, not +something externally visible as API. Can have cyclic dependences between +other components. Components roughly correspond to "xcode project" or +"B&I project" granularity at Apple. Can rebuild a "debug version" of a +subcomponent and drop it into an app without rebuilding the entire +world. + +**Source File**: Organizational unit within a component. + +In the trivial hello world example, the source file gets implicitly +dropped into a default component (since it doesn't have a component +declaration). The default component has settings that corresponds to an +executable. As the app grows and wants to start using sub- libraries, +the author would have to know about components. This ensures a simple +model for new people, because they don't need to know anything about +components until they want to define a library and stable APIs. + +We'll also eventually build tools to do things like: + +- Inspect and maintain dependence graphs between components + and subcomponents. +- Diff API \[semantically, not "by symbol" like 'nm'\] across versions + of products +- Provide code migration tools, like "rewrite rules" to update clients + that use obsoleted and removed API. +- Pure swift apps won't be able to use SPI (they just won't build), + but mixed swift/C apps could (through the C parts, similar to using + things like "extern int Z3fooi(int)" to access C++ mangled symbols + from C today). It will be straight-forward to write a binary + verifier that cross references the NM output with the manifest file + of the components it legitimately depends on. +- Lots of other cool stuff I'm sure. + +Anyway, that's the high-level thoughts and motivation, this is what I'm +proposing: + +Program structure +----------------- + +Programs and frameworks in swift consist of declarations (functions, +variables, types) that are (optionally) defined in possibly nested +namespaces, which are nested in a component, which are (optionally) +split into subcomponents. Components can also have associated resources +like images and plists, as well as code written in C/C++/ObjC. + +A "**Top Level Component**" (also referred to as "an ownership domain") +is a unit of code that is owned by a single organization and is updated +(shipped to customers) as a whole. Examples of different top-level +components are products like the swift standard libraries, Mac OS/X, +iOS, Xcode, iWork, and even small things like a theoretical third-party +Perforce plugin to Xcode. + +Components are explicitly declared, and these declarations can include: + +- whether the component should be built into a dylib or executable, or + is a subcomponent. +- the version of the component (which are used for "availability + macros" etc) +- an explicit list of dependences on other top-level components (whose + dependence graph is required to be acyclic) optionally with specific + versions: "I depend on swift standard libs 1.4 or later" +- a list of subcomponents that contribute to the component: "mac os + consists of appkit, coredata, …" +- a list of resource files and other stuff that makes up the framework +- A list of subdirectories to get source files out of (see filesystem + layout below) if the component is more that one directory full + of code. +- A list of any .c/.m/.cpp files that are built and linked into the + component, along with build flags etc. + +Top-Level Components define the top level of the namespace stack. This +means everything in the swift libraries are "swift.array.xyz", +everything in MacOS/X is "macosx.whatever". Thus you can't have naming +conflicts across components. + +**Namespaces** are for organization within a component, and are left up +to the developer to handle however they want. They will work similarly +to C++ namespaces and aren't described in detail here. For example, you +could have a macosx.coredata namespace that coredata drops all its stuff +into. + +Components can optionally be broken into a set of "**Subcomponents**", +which are organizational units within a top-level component. +Subcomponents exist to support extremely large components that have +multiple different teams contributing to a single large product. +Subcomponents are purely an implementation detail of top- level +components and have no runtime, naming/namespace, or other externally +visible artifacts that persist once the entire domain is built. If +version 1.0 of a domain is shipped, version 1.1 can completely reshuffle +the internal subcomponent organization without affecting its published +API or anything else a client can see. + +Subcomponents are explicitly declared, and these declarations can +include: + +- The component they belong to. +- The set of other (optionally versioned) top-level components they + depend on. +- The set of components (within the current top-level component) that + this subcomponent depends on. This dependence is an acyclic + dependence: "core data depends on foundation". +- A list of declarations they use within the current top-level + component that aren't provided by the subcomponents they explicitly + depend on. This is used to handle cyclic dependencies across + subcomponents within an ownership domain: for example: "libsystem + depends on libcompiler\_rt", however, "libcompiler\_rt depends on + 'func abort();' in libsystem". This preserves the acyclic + compilation order across components. +- A list of subdirectories to get source files out of (see filesystem + layout below) if the component is more that one directory full + of code. +- A list of any .c/.m/.cpp files that are linked into the component, + with build flags. + +**Source Files** and **Resources** make up a component. Swift source +files can include: + +- The component they belong to. +- Import declarations that affect their local scope lookups (similar + to java import statements) +- A set of declarations of variables, functions, types etc. +- C and other language files are just another kind of resource to + be built. + +**Declarations** of variables, functions and types are the meat of the +program, and populate source files. Declarations can be scoped to be +externally exported from the component (aka API), internal to the +component (aka SPI), local to a subcomponent (aka "visibility hidden", +the default), or local to the file (aka static). Top-level components +also have a simple runtime representation which is used to ensure that +reflection only returns API and decls within the current ownership +domain: "App's can't get at iOS SPI". + +**Executable expressions** can also be included at file scope (outside +other declarations). This global code is run at startup time (same as +static constructors), eliminating the need for "main". This +initialization code is correctly run bottom-up in the explicit +dependence graph. Order of initialization between multiple cyclicly +dependent files within a single component is not defined (and perhaps we +can make it be an outright error). + +File system layout and compiler UI +---------------------------------- + +The filesystem layout of a component is a directory with at least one +.swift file in it that has the same name as the directory. A common case +is that the component is a single directory with a bunch of .swift files +and resources in it. The "large component" case can break up its source +files and resources into subdirectories. + +Here is the minimal hello world example written as a proper app: + + myapp/ + myapp.swift + +You'd compile it like this: + + $ swift myapp + myapp compiled successfully! + +or: + + $ cd myapp + $ swift + myapp compiled successfully! + +and it would produce this filesystem layout: + + myapp/ + myapp.swift + products/ + myapp + myapp.manifest + buildcache/ + + +Here is a moderately complicated example of a library: + + mylib/ + mylib.swift + a.swift + b.swift + UserManual.html + subdir/ + c.swift + d.swift + e.png + +mylib.swift tells the compiler about your sub directories, resources, +how to process them, where to put them, etc. After compiling it you'd +keep your source files and get: + + mylib/ + products/ + mylib.dylib + mylib.manifest + e.png + docs/ + UserManual.html + buildcache/ + + +Swift compiler command line is very simple: "swift mylib" is enough for +most uses. For more complex use cases we'll support specifying paths to +search for components (similar to clang -F or -L) etc. We'll also +support a "clean" command that nukes buildcache/ and products/. + +The BuildCache directory holds object files, dependence information and +other stuff needed for incremental \[re\]builds within the component The +generated manifest file is used both the compiler when a clients lib/app +import mylib (it contains type information for all the stuff exported +from mylib) but also at runtime by the runtime library (e.g. for +reflection). It needs to be a fast-to-read but extensible format. + +What the build system does, how it works +---------------------------------------- + +Assuming that we're starting with an empty build cache, the build system +starts by parsing the mylib.swift file (the main file for the +directory). This file contains the component declaration. If this is a +subcomponent, the subcomponent declares which super-component it is in +(in which case, the super-component info is loaded). In either case, the +compiler verifies that all of the depended-on components are built, if +not, it goes off and recursively builds them before handling this one: +the component dependence graph is acyclic, and cycles are diagnosed +here. + +If this directory is a subcomponent (as opposed to a top-level +component), the subcomponent declaration has already been read. If this +subcomponent depends on any other components that are not up-to- date, +those are recursively rebuilt. Explicit subcomponent dependencies are +acyclic and cycles are diagnosed here. Now all depended-on top-level +components and subcomponents are built. + +Now the compiler parses each swift file into an AST. We'll keep the +swift grammar carefully factored to keep types and values distinct, so +it is possible to parse (but not fully typecheck) the files without +first reading "all the headers they depend on". This is important +because we want to allow arbitrary type and value cyclic dependences +between files in a component. As each file is parsed, the compiler +resolves as many intra-file references as it can, and ends up with a +list of (namespace qualified) types and values that are imported by the +file that are not satisfied by other components. This is the list of +things the file requires that some other files in the component provide. + +Now that the compiler has the full set of dependence information between +files in a component, it processes the files in strongly connected +component (SCC) order processing an SCC of dependent files at a time. +Given the entire SCC it is able to resolve values and types across the +files (without needing prototypes) and complete type checking. Assuming +type checking is successful (no errors) it generates code for each file +in the SCC, emits a .o file for them, and emits some extra metadata to +accelerate incremental builds. If there are .c files in the component, +they are compiled to .o files now (they are also described in the +component declaration). + +Once all of the source files are compiled into .o files, they are linked +into a final linked image (dylib or executable). At this point, a couple +of other random things are done: 1) metadata is checked to ensure that +any explicitly declared cyclic dependencies match the given and actual +prototype. 2) resources are copied or processed into the product +directory. 3) the explicit dependence graph is verified, extraneous +edges are warned about, missing edges are errors. + +In terms of implementation, this should be relatively straight- forward, +and is carefully layered to be memory efficient (e.g. only processing an +SCC at a time instead of an entire component) as well as highly parallel +for multicore machines. For incremental builds, we will have a huge win +because the fine-grained dependence information between .o files is +tracked and we know exactly what dependences to rebuild if anything +changes. The build cache will accelerate most of this, which will +eventually be a hybrid on-disk/in-memory data structure. + +The build system should be scalable enough for B&I to eventually do a +"swift macos" and have it do a full incremental (and parallel) build of +something the scale of Mac OS. Actually implementing this will obviously +be a big project that can happen as the installed base of swift code +grows. + +SDKs +---- + +The manifest file generated as a build product describes (among other +things) the full list of decls exported by the top-level component +(which includes their type information, not just symbol names). This +manifest file is used when a client builds against the component to type +check the client and ensure that its references are resolved. + +Because we have the version number as well as the full interface to the +component available in a consumable format is that we can build a SDK +generation tool. This tool would take manifest files for a set of +releases (e.g. iOS 4.0, 4.0.1, 4.0.2, 4.1, 4.1.1, 4.2) and build a +single SDK manifest which would have a mapping from symbol+type -> +version list that indicates what the versions a given symbol are +available in. This means that framework authors don't have to worry +about availability macros etc, it just naturally falls out of the +system. + +This tool can also produce warnings/errors about cases where API is in +version N but removed in version N+1, or when some declaration has an +invalid change (e.g. an argument added or something else "fragile"). +Blue sky idea: We could conceivable extend it so that the SDK manifest +file contains rewrite rules for obsolete APIs that the compiler could +automatically apply to upgrade user's source code. + +Future optimization opportunities +--------------------------------- + +The system has been carefully designed to allow fast builds at -O0 +(including keeping cached dependence information and the compiler around +in memory "across builds"), allowing a very incremental compilation +model and allowing carefully limited/understood cyclic dependencies +across components. However, we also care about really fast runtime +performance (better than our current system), and we should be able to +get that as well. + +There are several different possibilities to look at in the future: + +1. Components are a natural unit to do "link time" optimization. Since + the entire thing is shipped as a unit, we know that it is safe to + inline functions and analyze side effects within the bounds of + the component. This current LTO model should scale to the component + level, but we'd need new (more scalable/parallel and + memory efficient) approaches to optimize across the entire mac + os product. Processing components bottom-up within a large component + allows efficient context sensitive (and summary-based) analyzes, + like mod/ref, interprocedural constant prop, inlining, and + nocapture propagation. I expect nocapture to be specifically + important to get stuff on the stack instead of causing them to get + promoted to the heap all the time. +2. The dyld shared cache can be seen as an optimization across + components within the mac os top-level component. Though it has the + capability to include third party and other dylibs, in practice it + is rooted from a few key apps, so it doesn't get "everything" in + macos and it isn't used for other stuff (like xcode). The proposed + (but never implemented) "per-app shared cache" is a straight-forward + extension if this were based on optimizing across components. +3. There are a bunch of optimizations to take advantage of known + fragility levels for devirtualization, inlining, and other stuff + that I'm not going to describe here. Generalization of DaveZ's + positive/negative ivar/vtable idea. +4. The low level tools are already factored to be mostly object file + format independent. There is no reason that we need to keep using + actual macho .o files if it turns out to be inconvenient. We + obviously must keep around macho executables and dylibs. diff --git a/docs/proposals/archive/ProgramStructureAndCompilationModel.rst b/docs/proposals/archive/ProgramStructureAndCompilationModel.rst deleted file mode 100644 index 1f67481707cc1..0000000000000 --- a/docs/proposals/archive/ProgramStructureAndCompilationModel.rst +++ /dev/null @@ -1,385 +0,0 @@ -:orphan: - -.. @raise litre.TestsAreMissing -.. _ProgramStructureAndCompilationModel: - -Swift Program Structure and Compilation Model -============================================= - -.. warning:: This is a very early design document discussing the features of - a Swift build model and modules system. It should not be taken as a plan of - record. - -Commentary ----------- - -The C spec only describes things up to translation unit granularity: no -discussion of file system layout, build system, linking, runtime concepts of -code (dynamic libraries, executables, plugins), dependence between parts of a -program, Versioning + SDKs, human factors like management units, etc. It leaves -all of this up to implementors to sort out, and we got what the unix world -defined in the 60's and 70's with some minor stuff that could be shoehorned into -the old unix toolchain model without too much trouble. C also doesn't help with -resources (images etc), has a miserable incremental compilation model and many, -many, other issues. - -Swift should strive to make trivial programs really simple. Hello world should -just be something like:: - - print("hello world") - -while also acknowledging and strongly supporting the real world demands and -requirements that library implementors (hey, that's us!) face every day. In -particular, note how the language elements (described below) correspond directly -to the business and management reality of the world: - -**Ownership Domain / Top Level Component**: corresponds to a product that is -shipped as a unit (Mac OS/X, iWork, Xcode), is a collection of frameworks/dylibs -and resources. Only acyclic dependences between different domains is -allowed. There is some correlation in concept here to "umbrella headers" or -"dyld shared cache" though it isn't exact. - -**Namespace**: Organizational structure within a domain, similar to C++ or -Java. Programmers can use or abuse them however they wish. - -**Subcomponent**: corresponds to an individual team or management unit, is one -dylib + optional resources. All contributing source files and resources live in -one directory (with optional subdirs), and have a single "project file". Can -contribute to multiple namespaces. The division of a domain into components is -an implementation detail, not something externally visible as API. Can have -cyclic dependences between other components. Components roughly correspond to -"xcode project" or "B&I project" granularity at Apple. Can rebuild a "debug -version" of a subcomponent and drop it into an app without rebuilding the entire -world. - -**Source File**: Organizational unit within a component. - -In the trivial hello world example, the source file gets implicitly dropped into -a default component (since it doesn't have a component declaration). The default -component has settings that corresponds to an executable. As the app grows and -wants to start using sub- libraries, the author would have to know about -components. This ensures a simple model for new people, because they don't need -to know anything about components until they want to define a library and stable -APIs. - -We'll also eventually build tools to do things like: - -* Inspect and maintain dependence graphs between components and subcomponents. - -* Diff API [semantically, not "by symbol" like 'nm'] across versions of products - -* Provide code migration tools, like "rewrite rules" to update clients that use - obsoleted and removed API. - -* Pure swift apps won't be able to use SPI (they just won't build), but mixed - swift/C apps could (through the C parts, similar to using things like "extern - int Z3fooi(int)" to access C++ mangled symbols from C today). It will be - straight-forward to write a binary verifier that cross references the NM - output with the manifest file of the components it legitimately depends on. - -* Lots of other cool stuff I'm sure. - -Anyway, that's the high-level thoughts and motivation, this is what I'm -proposing: - -Program structure ------------------ - -Programs and frameworks in swift consist of declarations (functions, variables, -types) that are (optionally) defined in possibly nested namespaces, which are -nested in a component, which are (optionally) split into -subcomponents. Components can also have associated resources like images and -plists, as well as code written in C/C++/ObjC. - -A "**Top Level Component**" (also referred to as "an ownership domain") is a -unit of code that is owned by a single organization and is updated (shipped to -customers) as a whole. Examples of different top-level components are products -like the swift standard libraries, Mac OS/X, iOS, Xcode, iWork, and even small -things like a theoretical third-party Perforce plugin to Xcode. - -Components are explicitly declared, and these declarations can include: - -* whether the component should be built into a dylib or executable, or is a - subcomponent. - -* the version of the component (which are used for "availability macros" etc) - -* an explicit list of dependences on other top-level components (whose - dependence graph is required to be acyclic) optionally with specific versions: - "I depend on swift standard libs 1.4 or later" - -* a list of subcomponents that contribute to the component: "mac os consists of - appkit, coredata, …" - -* a list of resource files and other stuff that makes up the framework - -* A list of subdirectories to get source files out of (see filesystem layout - below) if the component is more that one directory full of code. - -* A list of any .c/.m/.cpp files that are built and linked into the component, - along with build flags etc. - -Top-Level Components define the top level of the namespace stack. This means -everything in the swift libraries are "swift.array.xyz", everything in MacOS/X -is "macosx.whatever". Thus you can't have naming conflicts across components. - -**Namespaces** are for organization within a component, and are left up to the -developer to handle however they want. They will work similarly to C++ -namespaces and aren't described in detail here. For example, you could have a -macosx.coredata namespace that coredata drops all its stuff into. - -Components can optionally be broken into a set of "**Subcomponents**", which are -organizational units within a top-level component. Subcomponents exist to -support extremely large components that have multiple different teams -contributing to a single large product. Subcomponents are purely an -implementation detail of top- level components and have no runtime, -naming/namespace, or other externally visible artifacts that persist once the -entire domain is built. If version 1.0 of a domain is shipped, version 1.1 can -completely reshuffle the internal subcomponent organization without affecting -its published API or anything else a client can see. - -Subcomponents are explicitly declared, and these declarations can include: - -* The component they belong to. - -* The set of other (optionally versioned) top-level components they depend on. - -* The set of components (within the current top-level component) that this - subcomponent depends on. This dependence is an acyclic dependence: "core data - depends on foundation". - -* A list of declarations they use within the current top-level component that - aren't provided by the subcomponents they explicitly depend on. This is used - to handle cyclic dependencies across subcomponents within an ownership domain: - for example: "libsystem depends on libcompiler_rt", however, "libcompiler_rt - depends on 'func abort();' in libsystem". This preserves the acyclic - compilation order across components. - -* A list of subdirectories to get source files out of (see filesystem layout - below) if the component is more that one directory full of code. - -* A list of any .c/.m/.cpp files that are linked into the component, with build - flags. - -**Source Files** and **Resources** make up a component. Swift source files can -include: - -* The component they belong to. - -* Import declarations that affect their local scope lookups (similar to java - import statements) - -* A set of declarations of variables, functions, types etc. - -* C and other language files are just another kind of resource to be built. - -**Declarations** of variables, functions and types are the meat of the program, -and populate source files. Declarations can be scoped to be externally exported -from the component (aka API), internal to the component (aka SPI), local to a -subcomponent (aka "visibility hidden", the default), or local to the file (aka -static). Top-level components also have a simple runtime representation which is -used to ensure that reflection only returns API and decls within the current -ownership domain: "App's can't get at iOS SPI". - -**Executable expressions** can also be included at file scope (outside other -declarations). This global code is run at startup time (same as static -constructors), eliminating the need for "main". This initialization code is -correctly run bottom-up in the explicit dependence graph. Order of -initialization between multiple cyclicly dependent files within a single -component is not defined (and perhaps we can make it be an outright error). - -File system layout and compiler UI ----------------------------------- - -The filesystem layout of a component is a directory with at least one .swift -file in it that has the same name as the directory. A common case is that the -component is a single directory with a bunch of .swift files and resources in -it. The "large component" case can break up its source files and resources into -subdirectories. - -Here is the minimal hello world example written as a proper app:: - - myapp/ - myapp.swift - -You'd compile it like this:: - - $ swift myapp - myapp compiled successfully! - -or:: - - $ cd myapp - $ swift - myapp compiled successfully! - -and it would produce this filesystem layout:: - - myapp/ - myapp.swift - products/ - myapp - myapp.manifest - buildcache/ - - -Here is a moderately complicated example of a library:: - - mylib/ - mylib.swift - a.swift - b.swift - UserManual.html - subdir/ - c.swift - d.swift - e.png - -mylib.swift tells the compiler about your sub directories, resources, how to -process them, where to put them, etc. After compiling it you'd keep your source -files and get:: - - mylib/ - products/ - mylib.dylib - mylib.manifest - e.png - docs/ - UserManual.html - buildcache/ - - -Swift compiler command line is very simple: "swift mylib" is enough for most -uses. For more complex use cases we'll support specifying paths to search for -components (similar to clang -F or -L) etc. We'll also support a "clean" command -that nukes buildcache/ and products/. - -The BuildCache directory holds object files, dependence information and other -stuff needed for incremental [re]builds within the component The generated -manifest file is used both the compiler when a clients lib/app import mylib (it -contains type information for all the stuff exported from mylib) but also at -runtime by the runtime library (e.g. for reflection). It needs to be a -fast-to-read but extensible format. - -What the build system does, how it works ----------------------------------------- - -Assuming that we're starting with an empty build cache, the build system starts -by parsing the mylib.swift file (the main file for the directory). This file -contains the component declaration. If this is a subcomponent, the subcomponent -declares which super-component it is in (in which case, the super-component info -is loaded). In either case, the compiler verifies that all of the depended-on -components are built, if not, it goes off and recursively builds them before -handling this one: the component dependence graph is acyclic, and cycles are -diagnosed here. - -If this directory is a subcomponent (as opposed to a top-level component), the -subcomponent declaration has already been read. If this subcomponent depends on -any other components that are not up-to- date, those are recursively -rebuilt. Explicit subcomponent dependencies are acyclic and cycles are diagnosed -here. Now all depended-on top-level components and subcomponents are built. - -Now the compiler parses each swift file into an AST. We'll keep the swift -grammar carefully factored to keep types and values distinct, so it is possible -to parse (but not fully typecheck) the files without first reading "all the -headers they depend on". This is important because we want to allow arbitrary -type and value cyclic dependences between files in a component. As each file is -parsed, the compiler resolves as many intra-file references as it can, and ends -up with a list of (namespace qualified) types and values that are imported by -the file that are not satisfied by other components. This is the list of things -the file requires that some other files in the component provide. - -Now that the compiler has the full set of dependence information between files -in a component, it processes the files in strongly connected component (SCC) -order processing an SCC of dependent files at a time. Given the entire SCC it is -able to resolve values and types across the files (without needing prototypes) -and complete type checking. Assuming type checking is successful (no errors) it -generates code for each file in the SCC, emits a .o file for them, and emits -some extra metadata to accelerate incremental builds. If there are .c files in -the component, they are compiled to .o files now (they are also described in the -component declaration). - -Once all of the source files are compiled into .o files, they are linked into a -final linked image (dylib or executable). At this point, a couple of other -random things are done: 1) metadata is checked to ensure that any explicitly -declared cyclic dependencies match the given and actual prototype. 2) resources -are copied or processed into the product directory. 3) the explicit dependence -graph is verified, extraneous edges are warned about, missing edges are errors. - -In terms of implementation, this should be relatively straight- forward, and is -carefully layered to be memory efficient (e.g. only processing an SCC at a time -instead of an entire component) as well as highly parallel for multicore -machines. For incremental builds, we will have a huge win because the -fine-grained dependence information between .o files is tracked and we know -exactly what dependences to rebuild if anything changes. The build cache will -accelerate most of this, which will eventually be a hybrid on-disk/in-memory -data structure. - -The build system should be scalable enough for B&I to eventually do a "swift -macos" and have it do a full incremental (and parallel) build of something the -scale of Mac OS. Actually implementing this will obviously be a big project that -can happen as the installed base of swift code grows. - -SDKs ----- - -The manifest file generated as a build product describes (among other things) -the full list of decls exported by the top-level component (which includes their -type information, not just symbol names). This manifest file is used when a -client builds against the component to type check the client and ensure that its -references are resolved. - -Because we have the version number as well as the full interface to the -component available in a consumable format is that we can build a SDK generation -tool. This tool would take manifest files for a set of releases (e.g. iOS 4.0, -4.0.1, 4.0.2, 4.1, 4.1.1, 4.2) and build a single SDK manifest which would have -a mapping from symbol+type -> version list that indicates what the versions a -given symbol are available in. This means that framework authors don't have to -worry about availability macros etc, it just naturally falls out of the system. - -This tool can also produce warnings/errors about cases where API is in version N -but removed in version N+1, or when some declaration has an invalid change -(e.g. an argument added or something else "fragile"). Blue sky idea: We could -conceivable extend it so that the SDK manifest file contains rewrite rules for -obsolete APIs that the compiler could automatically apply to upgrade user's -source code. - -Future optimization opportunities ---------------------------------- - -The system has been carefully designed to allow fast builds at -O0 (including -keeping cached dependence information and the compiler around in memory "across -builds"), allowing a very incremental compilation model and allowing carefully -limited/understood cyclic dependencies across components. However, we also care -about really fast runtime performance (better than our current system), and we -should be able to get that as well. - -There are several different possibilities to look at in the future: - -1. Components are a natural unit to do "link time" optimization. Since the - entire thing is shipped as a unit, we know that it is safe to inline - functions and analyze side effects within the bounds of the component. This - current LTO model should scale to the component level, but we'd need new - (more scalable/parallel and memory efficient) approaches to optimize across - the entire mac os product. Processing components bottom-up within a large - component allows efficient context sensitive (and summary-based) analyzes, - like mod/ref, interprocedural constant prop, inlining, and nocapture - propagation. I expect nocapture to be specifically important to get stuff on - the stack instead of causing them to get promoted to the heap all the time. - -2. The dyld shared cache can be seen as an optimization across components within - the mac os top-level component. Though it has the capability to include third - party and other dylibs, in practice it is rooted from a few key apps, so it - doesn't get "everything" in macos and it isn't used for other stuff (like - xcode). The proposed (but never implemented) "per-app shared cache" is a - straight-forward extension if this were based on optimizing across - components. - -3. There are a bunch of optimizations to take advantage of known fragility - levels for devirtualization, inlining, and other stuff that I'm not going to - describe here. Generalization of DaveZ's positive/negative ivar/vtable idea. - -4. The low level tools are already factored to be mostly object file format - independent. There is no reason that we need to keep using actual macho .o - files if it turns out to be inconvenient. We obviously must keep around macho - executables and dylibs. diff --git a/docs/proposals/archive/UnifiedFunctionSyntax.md b/docs/proposals/archive/UnifiedFunctionSyntax.md new file mode 100644 index 0000000000000..73680914f583c --- /dev/null +++ b/docs/proposals/archive/UnifiedFunctionSyntax.md @@ -0,0 +1,507 @@ +Unified Function Syntax via Selector Splitting +============================================== + +> **warning** +> +> This document was used in planning Swift 1.0; it has not been kept + +> up to date and does not describe the current or planned behavior of +> Swift. In particular, we experimented with preposition-based splitting +> and decided against it. + +Cocoa Selectors +--------------- + +A Cocoa selector is intended to convey what a method does or produces as +well as what its various arguments are. For example, `NSTableView` has +the following method: + + - (void)moveRowAtIndex:(NSInteger)oldIndex toIndex:(NSInteger)newIndex; + +Note that there are three pieces of information in the selector +`moveRowAtIndex:toIndex:`: + +1. What the method is doing ("moving a row"). +2. What the first argument is ("the index of the row we're moving"). +3. What the second argument is ("the index we're moving to"). + +However, there are only two selector pieces: "moveRowAtIndex" and +"toIndex". The first selector piece is conveying both \#1 and \#2, and +it reads well in English because the preposition "at" separates the +action (`moveRow`) from the first argument (`AtIndex`), while the second +selector piece conveys \#3. Cocoa conventions in this area are fairly +strong, where the first selector piece describes what the operation is +doing or produces, and well as what the first argument is, and +subsequent selector pieces describe the remaining arguments. + +Splitting Selectors at Prepositions +----------------------------------- + +When importing an Objective-C selector, split the first selector piece +into a base method name and a first argument name. The actual split will +occur just before the last preposition in the selector piece, using +camelCase word boundaries to identify words. The resulting method name +is: + + moveRow(atIndex:toIndex:) + +where `moveRow` is the base name, `atIndex` is the name of the first +argument (note that the 'a' has been automatically lowercased), and +`toIndex` is the name of the second argument. + +In the (fairly rare) case where there are two prepositions in the +initial selector, splitting at the last preposition improves the +likelihood of a better split, because the last prepositional phrase is +more likely to pertain to the first argument. For example, +`appendBezierPathWithArcFromPoint:toPoint:radius:` becomes: + + appendBezierPathWithArc(fromPoint:toPoint:radius:) + +If there are no prepositions within the first selector piece, the entire +first selector piece becomes the base name, and the first argument is +unnamed. For example `UIView`'s `insertSubview:atIndex:` becomes: + + insertSubview(_:atIndex:) + +where '\_' is a placeholder for an argument with no name. + +Calling Syntax +-------------- + +By splitting selectors into a base name and argument names, Swift's +keyword-argument calling syntax works naturally: + + tableView.moveRow(atIndex: i, toIndex: j) + view.insertSubview(someView, atIndex: i) + +The syntax generalizes naturally to global and local functions that have +no object argument, i.e.,: + + NSMakeRange(location: loc, length: len) + +assuming that we had argument names for C functions or a Swift overlay +that provided them. It also nicely handles cases where argument names +aren't available, e.g.,: + + NSMakeRange(loc, len) + +as well as variadic methods: + + NSString.string(withFormat: "%@ : %@", key, value) + +Declaration Syntax +------------------ + +The existing "selector-style" declaration syntax can be extended to +better support declaring functions with separate base names and first +argument names, i.e.: + + func moveRow atIndex(Int) toIndex(Int) + +However, this declaration looks very little like the call site, which +uses a parenthesized argument list, commas, and colons. Let's eliminate +the "selector-style" declaration syntax entirely. We can use the +existing ("tuple-style") declaration syntax to mirror the call syntax +directly: + + func moveRow(atIndex: Int, toIndex: Int) + +Now, sometimes the argument name that works well at the call site +doesn't work well for the body of the function. For example, splitting +the selector for `UIView`'s `contentHuggingPriorityForAxis:` results in: + + func contentHuggingPriority(forAxis: UILayoutConstraintAxis) -> UILayoutPriority + +The name `forAxis` works well at the call site, but not within the +function body. So, we allow one to specify the name of the parameter for +the body of the function: + + func contentHuggingPriority(forAxis axis: UILayoutConstraintAxis) -> UILayoutPriority { + // use 'axis' in the body + } + +One can use '\_' in either the argument or parameter name position to +specify that there is no name. For example: + + func f(_ a: Int) // no argument name; parameter name is 'a' + func g(b _: Int) // argument name is 'b'; no parameter name + +The first function doesn't support keyword arguments; it is what an +imported C or C++ function would use. The second function supports a +keyword argument (`b`), but the parameter is not named (and therefore +cannot be used) within the body. The second form is fairly uncommon, and +will presumably only to be used for backward compatibility. + +Method Names +------------ + +The name of a method in this scheme is determined by the base name and +the names of each of the arguments, and is written as: + + basename(param1:param2:param3:) + +to mirror the form of declarations and calls, with types, arguments, and +commas omitted. In code, one can refer to the name of a function just by +its basename, if the context provides enough information to uniquely +determine the method. For example, when uncurrying a method reference to +a variable of specified type: + + let f: (UILayoutConstraintAxis) -> UILayoutPriority = view.contentHuggingPriority + +To refer to the complete method name, place the method name in +backticks, as in this reference to an optional method in a delegate: + + if let method = delegate.`tableView(_:viewForTableColumn:row:)` { + // ... + } + +Initializers +------------ + +Objective-C `init` methods correspond to initializers in Swift. Swift +splits the selector name after the `init`. For example, `NSView`'s +`initWithFrame:` method becomes the initializer: + + init(withFrame: NSRect) + +There is a degenerate case here where the `init` method has additional +words following `init`, but there is no argument with which to associate +the information, such as with `initForIncrementalLoad`. This is +currently handled by adding an empty tuple parameter to store the name, +i.e.: + + init(forIncrementalLoad:()) + +which requires the somewhat unfortunate initialization syntax: + + NSBitmapImageRep(forIncrementalLoad:()) + +Fortunately, this is a relatively isolated problem: Cocoa and Cocoa +Touch contain only four selectors of this form: + + initForIncrementalLoad + initListDescriptor + initRecordDescriptor + initToMemory + +With a number that small, it's easy enough to provide overlays. + +Handling Poor Mappings +---------------------- + +The split-at-last-preposition heuristic works well for a significant +number of selectors, but it is not perfect. Therefore, we will introduce +an attribute into Objective-C that allows one to specify the Swift +method name for that Objective-C API. For example, by default, the +`NSURL` method `+bookmarkDataWithContentsOfURL:error:` will come into +Swift as: + + class func bookmarkDataWithContents(ofURL bookmarkFileURL: NSURL, inout error: NSError) -> NSData + +However, one can provide a different mapping with the `method_name` +attribute: + + + (NSData *)bookmarkDataWithContentsOfURL:(NSURL *)bookmarkFileURL error:(NSError **)error __attribute__((method_name(bookmarkData(withContentsOfURL:error:)))) + +This attribute specifies the Swift method name corresponding to that +selector. Presumably, the `method_name` attribute will be wrapped in a +macro supplied by Foundation, i.e.,: + + #define NS_METHOD_NAME(Name) __attribute__((method_name(Name))) + +For 1.0, it is not feasible to mark up the Objective-C headers in the +various SDKs. Therefore, the compiler will contain a list of mapping +from Objective-C selectors to Swift method names. Post-1.0, we can +migrate these mappings to the headers. + +A mapping in the other direction is also important, allowing one to +associate a specific Objective-C selector with a method. For example, a +Boolean property: + + var enabled: Bool { + @objc(isEnabled) get { + // ... + } + + set { + // ... + } + } + +Optionality and Ordering of Keyword Arguments +--------------------------------------------- + +A number of programming languages have keyword arguments in one form or +another, including Ada, C\#, Fortran 95, Lua, OCaml, Perl 6, Python, and +Ruby. Objective-C and Smalltalk's use of selectors is roughly +equivalent, in the sense that the arguments get names. The languages +with keyword arguments (but not Objective-C and Smalltalk) all allow +re-ordering of arguments at the call site, and many allow one to provide +arguments positionally without their associated name at the call site. +However, Cocoa APIs were designed based on the understanding that they +would not be re-ordered, and the sentence structure of some selectors +depends on that. To that end, a new attribute `call_arguments(strict)` +can be placed on any function and indicates that keyword arguments are +required and cannot be reordered in calls to that function, i.e.: + + @call_arguments(strict) + func moveRow(atIndex:Int, toIndex:Int) + +Swift's Objective-C importer will automatically add this to all imported +Objective-C methods, so that Cocoa APIs will retain their sentence +structure. + +Removing `with` and `for` from Argument Names +--------------------------------------------- + +The prepositions `with` and `for` are commonly used in the first +selector piece to separate the action or result of a method from the +first argument, but don't themselves convey much information at either +the call or declaration site. For example, `NSColor`'s +`colorWithRed:green:blue:alpha:` is called as: + + NSColor.color(withRed: 0.5, green: 0.5, blue: 0.5, alpha: 1.0) + +The `with` in this case feels spurious and makes `withRed` feel out of +sync with `green`, `blue`, and `alpha`. Therefore, we will remove the +`with` (or `for`) from any argument name, so that this call becomes: + + NSColor.color(red: 0.5, green: 0.5, blue: 0.5, alpha: 1.0) + +In addition to improving the call site, this eliminates the need to +rename parameters as often at the declaration site, i.e., this: + + class func color(withRed red: CGFloat, green: CGFloat, blue: CGFloat, alpha: CGFloat) -> NSColor + +becomes: + + class func color(red: CGFloat, green: CGFloat, blue: CGFloat, alpha: CGFloat) -> NSColor + +Note that we only perform this removal for `with` and `for`; other +prepositions tend to have important meaning associated with them, and +are therefore not removed. For example, consider calls to the `NSImage` +method `-drawInRect:fromRect:operation:fraction:` with the leading +prepositions retained and removed, respectively: + + image.draw(inRect: x, fromRect: x, operation: op, fraction: 0.5) + image.draw(rect: x, rect: y, operation: op, fraction: 0.5) + +Here, dropping the leading prepositions is actively harmful, because +we've lost the directionality provided by `in` and `from` in the first +two arguments. `with` and `for` do not have this problem. + +The second concern with dropping `with` and `for` is that we need to +either specify or infer the prepositions when declaring a method. For +example, consider the following initializer: + + init(frame: CGRect) + +How would the compiler know to insert the preposition "with" into the +name when computing the selector, so that this maps to `initWithFrame:`? +In many cases, where we're overriding a method or initializer from a +superclass or we are implementing a method to conform to a protocol, the +selector can be deduced from method/initializer in the superclass or +protocol. In those cases where new API is being defined in Swift where +the selector requires a preposition, one would use the `objc` attribute +with a selector: + + @objc(initWithFrame:) + init(frame: CGRect) + +Imported Objective-C methods would have the appropriate `objc` attribute +attached to them automatically. + +Which Prepositions? +------------------- + +English has a large number of prepositions, and many of those words also +have other rules as adjectives, adverbs, and so on. The following list, +taken from [The English +Club](http://www.englishclub.com/grammar/prepositions-list.htm), with +poetic, archaic, and non-US forms removed, provided the starting point +for the list of prepositions used in splitting. The **bolded** +prepositions are used to split; notes indicate whether Cocoa uses this +preposition as a preposition in any of its selectors, as well as any +special circumstances that affect inclusion or exclusion from the list. + ++-------------------+------------+------------+-------------------------------+ +| Preposition | In Cocoa? | Dropped? | > Notes | ++-------------------+------------+------------+-------------------------------+ +| Aboard | No | | | ++-------------------+------------+------------+-------------------------------+ +| About | No\* | | Used as an adjective | ++-------------------+------------+------------+-------------------------------+ +| **Above** | Yes | > No | | ++-------------------+------------+------------+-------------------------------+ +| Across | No | | | ++-------------------+------------+------------+-------------------------------+ +| **After** | Yes | > No | | ++-------------------+------------+------------+-------------------------------+ +| Against | Yes\* | | Misleading when split | ++-------------------+------------+------------+-------------------------------+ +| **Along** | Yes | > No | | ++-------------------+------------+------------+-------------------------------+ +| **Alongside** | Yes | > No | | ++-------------------+------------+------------+-------------------------------+ +| Amid | No | | | ++-------------------+------------+------------+-------------------------------+ +| Among | No | | | ++-------------------+------------+------------+-------------------------------+ +| Anti | No\* | | Used as an adjective | ++-------------------+------------+------------+-------------------------------+ +| Around | No | | | ++-------------------+------------+------------+-------------------------------+ +| **As** | Yes | No | | ++-------------------+------------+------------+-------------------------------+ +| Astride | No | | | ++-------------------+------------+------------+-------------------------------+ +| **At** | Yes | No | | ++-------------------+------------+------------+-------------------------------+ +| Bar | No\* | | Used as a noun | ++-------------------+------------+------------+-------------------------------+ +| Barring | No | | | ++-------------------+------------+------------+-------------------------------+ +| **Before** | Yes | No | | ++-------------------+------------+------------+-------------------------------+ +| Behind | No | | | ++-------------------+------------+------------+-------------------------------+ +| **Below** | Yes | No | | ++-------------------+------------+------------+-------------------------------+ +| Beneath | No | | | ++-------------------+------------+------------+-------------------------------+ +| Beside | No | | | ++-------------------+------------+------------+-------------------------------+ +| Besides | No | | | ++-------------------+------------+------------+-------------------------------+ +| Between | Yes | | Not amenable to parameters | ++-------------------+------------+------------+-------------------------------+ +| Beyond | No | | | ++-------------------+------------+------------+-------------------------------+ +| But | No | | | ++-------------------+------------+------------+-------------------------------+ +| **By** | Yes | No | | ++-------------------+------------+------------+-------------------------------+ +| Circa | No | | | ++-------------------+------------+------------+-------------------------------+ +| Concerning | No | | | ++-------------------+------------+------------+-------------------------------+ +| Considering | No | | | ++-------------------+------------+------------+-------------------------------+ +| Counting | No\* | | Used as an adjective | ++-------------------+------------+------------+-------------------------------+ +| Cum | No | | | ++-------------------+------------+------------+-------------------------------+ +| Despite | No | | | ++-------------------+------------+------------+-------------------------------+ +| Down | No\* | | Used as a noun | ++-------------------+------------+------------+-------------------------------+ +| During | Yes\* | | Misleading when split | ++-------------------+------------+------------+-------------------------------+ +| Except | No | | | ++-------------------+------------+------------+-------------------------------+ +| Excepting | No | | | ++-------------------+------------+------------+-------------------------------+ +| Excluding | No | | | ++-------------------+------------+------------+-------------------------------+ +| **Following** | Yes | No | | ++-------------------+------------+------------+-------------------------------+ +| **For** | Yes | **Yes** | | ++-------------------+------------+------------+-------------------------------+ +| **From** | Yes | No | | ++-------------------+------------+------------+-------------------------------+ +| **Given** | Yes\* | No | Never splits a slector | ++-------------------+------------+------------+-------------------------------+ +| **In** | Yes | No | | ++-------------------+------------+------------+-------------------------------+ +| **Including** | Yes\* | No | Never splits a selector | ++-------------------+------------+------------+-------------------------------+ +| **Inside** | Yes | No | | ++-------------------+------------+------------+-------------------------------+ +| **Into** | Yes | No | | ++-------------------+------------+------------+-------------------------------+ +| Less | No\* | | Always "less than" | ++-------------------+------------+------------+-------------------------------+ +| Like | Yes\* | | Misleading when split | ++-------------------+------------+------------+-------------------------------+ +| Minus | No | | | ++-------------------+------------+------------+-------------------------------+ +| Near | No | | | ++-------------------+------------+------------+-------------------------------+ +| Notwithstanding | No | | | ++-------------------+------------+------------+-------------------------------+ +| **Of** | Yes | No | | ++-------------------+------------+------------+-------------------------------+ +| Off | No\* | | Used as a noun | ++-------------------+------------+------------+-------------------------------+ +| **On** | Yes | No | | ++-------------------+------------+------------+-------------------------------+ +| Onto | No | | | ++-------------------+------------+------------+-------------------------------+ +| Opposite | No | | | ++-------------------+------------+------------+-------------------------------+ +| Out | No\* | | Used as an adverb | ++-------------------+------------+------------+-------------------------------+ +| Outside | Yes\* | | Misleading when split | ++-------------------+------------+------------+-------------------------------+ +| Over | No\* | | Used as an adverb | ++-------------------+------------+------------+-------------------------------+ +| Past | No | | | ++-------------------+------------+------------+-------------------------------+ +| Pending | No\* | | Used as an adjective | ++-------------------+------------+------------+-------------------------------+ +| Per | Yes\* | | Misleading to split | ++-------------------+------------+------------+-------------------------------+ +| Plus | No | | Used as an adjective | ++-------------------+------------+------------+-------------------------------+ +| Pro | No | | | ++-------------------+------------+------------+-------------------------------+ +| Regarding | No | | | ++-------------------+------------+------------+-------------------------------+ +| Respecting | No | | | ++-------------------+------------+------------+-------------------------------+ +| Round | No | | | ++-------------------+------------+------------+-------------------------------+ +| Save | No\* | | Used as adjective, verb | ++-------------------+------------+------------+-------------------------------+ +| Saving | No\* | | Used as adjective | ++-------------------+------------+------------+-------------------------------+ +| **Since** | Yes | No | | ++-------------------+------------+------------+-------------------------------+ +| Than | No\* | | Always "greater than" | ++-------------------+------------+------------+-------------------------------+ +| Through | Yes\* | | Misleading when split | ++-------------------+------------+------------+-------------------------------+ +| Throughout | No | | | ++-------------------+------------+------------+-------------------------------+ +| **To** | Yes | No | | ++-------------------+------------+------------+-------------------------------+ +| Toward | No | | | ++-------------------+------------+------------+-------------------------------+ +| Towards | No | | | ++-------------------+------------+------------+-------------------------------+ +| Under | No | | | ++-------------------+------------+------------+-------------------------------+ +| Underneath | No | | | ++-------------------+------------+------------+-------------------------------+ +| Unlike | No | | | ++-------------------+------------+------------+-------------------------------+ +| **Until** | Yes | No | | ++-------------------+------------+------------+-------------------------------+ +| Unto | No | | | ++-------------------+------------+------------+-------------------------------+ +| Up | No\* | | Used as adjective | ++-------------------+------------+------------+-------------------------------+ +| Upon | Yes\* | | Misleading when split | ++-------------------+------------+------------+-------------------------------+ +| Versus | No | | | ++-------------------+------------+------------+-------------------------------+ +| **Via** | Yes | No | | ++-------------------+------------+------------+-------------------------------+ +| **With** | Yes | **Yes** | | ++-------------------+------------+------------+-------------------------------+ +| **Within** | Yes | No | | ++-------------------+------------+------------+-------------------------------+ +| Without | Yes\* | | Misleading when split | ++-------------------+------------+------------+-------------------------------+ +| Worth | No | | | ++-------------------+------------+------------+-------------------------------+ diff --git a/docs/proposals/archive/UnifiedFunctionSyntax.rst b/docs/proposals/archive/UnifiedFunctionSyntax.rst deleted file mode 100644 index 5fa8e9a869bc0..0000000000000 --- a/docs/proposals/archive/UnifiedFunctionSyntax.rst +++ /dev/null @@ -1,506 +0,0 @@ -:orphan: - -Unified Function Syntax via Selector Splitting -============================================== - -.. warning:: This document was used in planning Swift 1.0; it has not been kept - up to date and does not describe the current or planned behavior of Swift. In - particular, we experimented with preposition-based splitting and decided - against it. - -.. contents:: - -Cocoa Selectors ---------------- -A Cocoa selector is intended to convey what a method does or produces -as well as what its various arguments are. For example, -``NSTableView`` has the following method:: - - - (void)moveRowAtIndex:(NSInteger)oldIndex toIndex:(NSInteger)newIndex; - -Note that there are three pieces of information in the selector -``moveRowAtIndex:toIndex:``: - -1. What the method is doing ("moving a row"). -2. What the first argument is ("the index of the row we're moving"). -3. What the second argument is ("the index we're moving to"). - -However, there are only two selector pieces: "moveRowAtIndex" and -"toIndex". The first selector piece is conveying both #1 and #2, and -it reads well in English because the preposition "at" separates the -action (``moveRow``) from the first argument (``AtIndex``), while the -second selector piece conveys #3. Cocoa conventions in this area are -fairly strong, where the first selector piece describes what the -operation is doing or produces, and well as what the first argument -is, and subsequent selector pieces describe the remaining arguments. - -Splitting Selectors at Prepositions ------------------------------------ -When importing an Objective-C selector, split the first selector piece -into a base method name and a first argument name. The actual split -will occur just before the last preposition in the selector piece, -using camelCase word boundaries to identify words. The resulting -method name is:: - - moveRow(atIndex:toIndex:) - -where ``moveRow`` is the base name, ``atIndex`` is the name of the -first argument (note that the 'a' has been automatically lowercased), -and ``toIndex`` is the name of the second argument. - -In the (fairly rare) case where there are two prepositions in the -initial selector, splitting at the last preposition improves the -likelihood of a better split, because the last prepositional phrase is -more likely to pertain to the first argument. For example, -``appendBezierPathWithArcFromPoint:toPoint:radius:`` becomes:: - - appendBezierPathWithArc(fromPoint:toPoint:radius:) - -If there are no prepositions within the first selector piece, the -entire first selector piece becomes the base name, and the first -argument is unnamed. For example ``UIView``'s -``insertSubview:atIndex:`` becomes:: - - insertSubview(_:atIndex:) - -where '_' is a placeholder for an argument with no name. - -Calling Syntax --------------- -By splitting selectors into a base name and argument names, Swift's -keyword-argument calling syntax works naturally:: - - tableView.moveRow(atIndex: i, toIndex: j) - view.insertSubview(someView, atIndex: i) - -The syntax generalizes naturally to global and local functions that -have no object argument, i.e.,:: - - NSMakeRange(location: loc, length: len) - -assuming that we had argument names for C functions or a Swift overlay -that provided them. It also nicely handles cases where argument names -aren't available, e.g.,:: - - NSMakeRange(loc, len) - -as well as variadic methods:: - - NSString.string(withFormat: "%@ : %@", key, value) - -Declaration Syntax ------------------- -The existing "selector-style" declaration syntax can be extended to -better support declaring functions with separate base names and first -argument names, i.e.:: - - func moveRow atIndex(Int) toIndex(Int) - -However, this declaration looks very little like the call site, which -uses a parenthesized argument list, commas, and colons. Let's -eliminate the "selector-style" declaration syntax entirely. We can use -the existing ("tuple-style") declaration syntax to mirror the call -syntax directly:: - - func moveRow(atIndex: Int, toIndex: Int) - -Now, sometimes the argument name that works well at the call site -doesn't work well for the body of the function. For example, splitting -the selector for ``UIView``'s ``contentHuggingPriorityForAxis:`` -results in:: - - func contentHuggingPriority(forAxis: UILayoutConstraintAxis) -> UILayoutPriority - -The name ``forAxis`` works well at the call site, but not within the -function body. So, we allow one to specify the name of the parameter -for the body of the function:: - - func contentHuggingPriority(forAxis axis: UILayoutConstraintAxis) -> UILayoutPriority { - // use 'axis' in the body - } - -One can use '_' in either the argument or parameter name position to -specify that there is no name. For example:: - - func f(_ a: Int) // no argument name; parameter name is 'a' - func g(b _: Int) // argument name is 'b'; no parameter name - -The first function doesn't support keyword arguments; it is what an -imported C or C++ function would use. The second function supports a -keyword argument (``b``), but the parameter is not named (and -therefore cannot be used) within the body. The second form is fairly -uncommon, and will presumably only to be used for backward -compatibility. - -Method Names ------------- -The name of a method in this scheme is determined by the base name and -the names of each of the arguments, and is written as:: - - basename(param1:param2:param3:) - -to mirror the form of declarations and calls, with types, arguments, -and commas omitted. In code, one can refer to the name of a function -just by its basename, if the context provides enough information to -uniquely determine the method. For example, when uncurrying a method -reference to a variable of specified type:: - - let f: (UILayoutConstraintAxis) -> UILayoutPriority = view.contentHuggingPriority - -To refer to the complete method name, place the method name in -backticks, as in this reference to an optional method in a delegate:: - - if let method = delegate.`tableView(_:viewForTableColumn:row:)` { - // ... - } - -Initializers ------------- -Objective-C ``init`` methods correspond to initializers in -Swift. Swift splits the selector name after the ``init``. For example, -``NSView``'s ``initWithFrame:`` method becomes the initializer:: - - init(withFrame: NSRect) - -There is a degenerate case here where the ``init`` method has -additional words following ``init``, but there is no argument with -which to associate the information, such as with -``initForIncrementalLoad``. This is currently handled by adding an -empty tuple parameter to store the name, i.e.:: - - init(forIncrementalLoad:()) - -which requires the somewhat unfortunate initialization syntax:: - - NSBitmapImageRep(forIncrementalLoad:()) - -Fortunately, this is a relatively isolated problem: Cocoa and Cocoa -Touch contain only four selectors of this form:: - - initForIncrementalLoad - initListDescriptor - initRecordDescriptor - initToMemory - -With a number that small, it's easy enough to provide overlays. - -Handling Poor Mappings ----------------------- -The split-at-last-preposition heuristic works well for a significant -number of selectors, but it is not perfect. Therefore, we will -introduce an attribute into Objective-C that allows one to specify the -Swift method name for that Objective-C API. For example, by default, -the ``NSURL`` method ``+bookmarkDataWithContentsOfURL:error:`` will -come into Swift as:: - - class func bookmarkDataWithContents(ofURL bookmarkFileURL: NSURL, inout error: NSError) -> NSData - -However, one can provide a different mapping with the ``method_name`` -attribute:: - - + (NSData *)bookmarkDataWithContentsOfURL:(NSURL *)bookmarkFileURL error:(NSError **)error __attribute__((method_name(bookmarkData(withContentsOfURL:error:)))) - -This attribute specifies the Swift method name corresponding to that -selector. Presumably, the ``method_name`` attribute will be wrapped in -a macro supplied by Foundation, i.e.,:: - - #define NS_METHOD_NAME(Name) __attribute__((method_name(Name))) - -For 1.0, it is not feasible to mark up the Objective-C headers in the -various SDKs. Therefore, the compiler will contain a list of mapping -from Objective-C selectors to Swift method names. Post-1.0, we can -migrate these mappings to the headers. - -A mapping in the other direction is also important, allowing one to -associate a specific Objective-C selector with a method. For example, -a Boolean property:: - - var enabled: Bool { - @objc(isEnabled) get { - // ... - } - - set { - // ... - } - } - -Optionality and Ordering of Keyword Arguments ---------------------------------------------- -A number of programming languages have keyword arguments in one form -or another, including Ada, C#, Fortran 95, Lua, OCaml, -Perl 6, Python, and Ruby. Objective-C and Smalltalk's use of selectors -is roughly equivalent, in the sense that the arguments get names. -The languages with keyword arguments (but not Objective-C and -Smalltalk) all allow re-ordering of -arguments at the call site, and many allow one to -provide arguments positionally without their associated name at the -call site. However, Cocoa APIs were designed based on the -understanding that they would not be re-ordered, and the sentence -structure of some selectors depends on that. To that end, a new -attribute ``call_arguments(strict)`` can be placed on any function and -indicates that keyword arguments are required and cannot be reordered -in calls to that function, i.e.:: - - @call_arguments(strict) - func moveRow(atIndex:Int, toIndex:Int) - -Swift's Objective-C importer will automatically add this to all -imported Objective-C methods, so that Cocoa APIs will retain their -sentence structure. - -Removing ``with`` and ``for`` from Argument Names -------------------------------------------------- -The prepositions ``with`` and ``for`` are commonly used in the first -selector piece to separate the action or result of a method from the -first argument, but don't themselves convey much information at either -the call or declaration site. For example, ``NSColor``'s -``colorWithRed:green:blue:alpha:`` is called as:: - - NSColor.color(withRed: 0.5, green: 0.5, blue: 0.5, alpha: 1.0) - -The ``with`` in this case feels spurious and makes ``withRed`` feel -out of sync with ``green``, ``blue``, and ``alpha``. Therefore, we -will remove the ``with`` (or ``for``) from any argument name, so that -this call becomes:: - - NSColor.color(red: 0.5, green: 0.5, blue: 0.5, alpha: 1.0) - -In addition to improving the call site, this eliminates the need to -rename parameters as often at the declaration site, i.e., this:: - - class func color(withRed red: CGFloat, green: CGFloat, blue: CGFloat, alpha: CGFloat) -> NSColor - -becomes:: - - class func color(red: CGFloat, green: CGFloat, blue: CGFloat, alpha: CGFloat) -> NSColor - -Note that we only perform this removal for ``with`` and ``for``; other -prepositions tend to have important meaning associated with them, and -are therefore not removed. For example, consider calls to the -``NSImage`` method ``-drawInRect:fromRect:operation:fraction:`` with -the leading prepositions retained and removed, respectively:: - - image.draw(inRect: x, fromRect: x, operation: op, fraction: 0.5) - image.draw(rect: x, rect: y, operation: op, fraction: 0.5) - -Here, dropping the leading prepositions is actively harmful, because -we've lost the directionality provided by ``in`` and ``from`` in the -first two arguments. ``with`` and ``for`` do not have this problem. - -The second concern with dropping ``with`` and ``for`` is that we need -to either specify or infer the prepositions when declaring a -method. For example, consider the following initializer:: - - init(frame: CGRect) - -How would the compiler know to insert the preposition "with" into the -name when computing the selector, so that this maps to -``initWithFrame:``? In many cases, where we're overriding a method or -initializer from a superclass or we are implementing a method to conform -to a protocol, the selector can be deduced from method/initializer in -the superclass or protocol. In those cases where new API is being -defined in Swift where the selector requires a preposition, one would -use the ``objc`` attribute with a selector:: - - @objc(initWithFrame:) - init(frame: CGRect) - -Imported Objective-C methods would have the appropriate ``objc`` -attribute attached to them automatically. - -Which Prepositions? -------------------- - -English has a large number of prepositions, and many of those words -also have other rules as adjectives, adverbs, and so on. The following -list, taken from `The English Club`_, with poetic, archaic, and non-US -forms removed, provided the starting point for the list of -prepositions used in splitting. The **bolded** prepositions are used -to split; notes indicate whether Cocoa uses this preposition as a -preposition in any of its selectors, as well as any special -circumstances that affect inclusion or exclusion from the list. - -+----------------+---------+---------+----------------------------+ -|Preposition |In Cocoa?|Dropped? | Notes | -+----------------+---------+---------+----------------------------+ -| Aboard | No | | | -+----------------+---------+---------+----------------------------+ -| About | No* | | Used as an adjective | -+----------------+---------+---------+----------------------------+ -| **Above** | Yes | No | | -+----------------+---------+---------+----------------------------+ -| Across | No | | | -+----------------+---------+---------+----------------------------+ -| **After** | Yes | No | | -+----------------+---------+---------+----------------------------+ -| Against | Yes* | | Misleading when split | -+----------------+---------+---------+----------------------------+ -| **Along** | Yes | No | | -+----------------+---------+---------+----------------------------+ -| **Alongside** | Yes | No | | -+----------------+---------+---------+----------------------------+ -| Amid | No | | | -+----------------+---------+---------+----------------------------+ -| Among | No | | | -+----------------+---------+---------+----------------------------+ -| Anti | No* | | Used as an adjective | -+----------------+---------+---------+----------------------------+ -| Around | No | | | -+----------------+---------+---------+----------------------------+ -| **As** | Yes | No | | -+----------------+---------+---------+----------------------------+ -| Astride | No | | | -+----------------+---------+---------+----------------------------+ -| **At** | Yes | No | | -+----------------+---------+---------+----------------------------+ -| Bar | No* | | Used as a noun | -+----------------+---------+---------+----------------------------+ -| Barring | No | | | -+----------------+---------+---------+----------------------------+ -| **Before** | Yes | No | | -+----------------+---------+---------+----------------------------+ -| Behind | No | | | -+----------------+---------+---------+----------------------------+ -| **Below** | Yes | No | | -+----------------+---------+---------+----------------------------+ -| Beneath | No | | | -+----------------+---------+---------+----------------------------+ -| Beside | No | | | -+----------------+---------+---------+----------------------------+ -| Besides | No | | | -+----------------+---------+---------+----------------------------+ -| Between | Yes | | Not amenable to parameters | -+----------------+---------+---------+----------------------------+ -| Beyond | No | | | -+----------------+---------+---------+----------------------------+ -| But | No | | | -+----------------+---------+---------+----------------------------+ -| **By** | Yes | No | | -+----------------+---------+---------+----------------------------+ -| Circa | No | | | -+----------------+---------+---------+----------------------------+ -| Concerning | No | | | -+----------------+---------+---------+----------------------------+ -| Considering | No | | | -+----------------+---------+---------+----------------------------+ -| Counting | No* | | Used as an adjective | -+----------------+---------+---------+----------------------------+ -| Cum | No | | | -+----------------+---------+---------+----------------------------+ -| Despite | No | | | -+----------------+---------+---------+----------------------------+ -| Down | No* | | Used as a noun | -+----------------+---------+---------+----------------------------+ -| During | Yes* | | Misleading when split | -+----------------+---------+---------+----------------------------+ -| Except | No | | | -+----------------+---------+---------+----------------------------+ -| Excepting | No | | | -+----------------+---------+---------+----------------------------+ -| Excluding | No | | | -+----------------+---------+---------+----------------------------+ -| **Following** | Yes | No | | -+----------------+---------+---------+----------------------------+ -| **For** | Yes | **Yes** | | -+----------------+---------+---------+----------------------------+ -| **From** | Yes | No | | -+----------------+---------+---------+----------------------------+ -| **Given** | Yes* | No | Never splits a slector | -+----------------+---------+---------+----------------------------+ -| **In** | Yes | No | | -+----------------+---------+---------+----------------------------+ -| **Including** | Yes* | No | Never splits a selector | -+----------------+---------+---------+----------------------------+ -| **Inside** | Yes | No | | -+----------------+---------+---------+----------------------------+ -| **Into** | Yes | No | | -+----------------+---------+---------+----------------------------+ -| Less | No* | | Always "less than" | -+----------------+---------+---------+----------------------------+ -| Like | Yes* | | Misleading when split | -+----------------+---------+---------+----------------------------+ -| Minus | No | | | -+----------------+---------+---------+----------------------------+ -| Near | No | | | -+----------------+---------+---------+----------------------------+ -| Notwithstanding| No | | | -+----------------+---------+---------+----------------------------+ -| **Of** | Yes | No | | -+----------------+---------+---------+----------------------------+ -| Off | No* | | Used as a noun | -+----------------+---------+---------+----------------------------+ -| **On** | Yes | No | | -+----------------+---------+---------+----------------------------+ -| Onto | No | | | -+----------------+---------+---------+----------------------------+ -| Opposite | No | | | -+----------------+---------+---------+----------------------------+ -| Out | No* | | Used as an adverb | -+----------------+---------+---------+----------------------------+ -| Outside | Yes* | | Misleading when split | -+----------------+---------+---------+----------------------------+ -| Over | No* | | Used as an adverb | -+----------------+---------+---------+----------------------------+ -| Past | No | | | -+----------------+---------+---------+----------------------------+ -| Pending | No* | | Used as an adjective | -+----------------+---------+---------+----------------------------+ -| Per | Yes* | | Misleading to split | -+----------------+---------+---------+----------------------------+ -| Plus | No | | Used as an adjective | -+----------------+---------+---------+----------------------------+ -| Pro | No | | | -+----------------+---------+---------+----------------------------+ -| Regarding | No | | | -+----------------+---------+---------+----------------------------+ -| Respecting | No | | | -+----------------+---------+---------+----------------------------+ -| Round | No | | | -+----------------+---------+---------+----------------------------+ -| Save | No* | | Used as adjective, verb | -+----------------+---------+---------+----------------------------+ -| Saving | No* | | Used as adjective | -+----------------+---------+---------+----------------------------+ -| **Since** | Yes | No | | -+----------------+---------+---------+----------------------------+ -| Than | No* | | Always "greater than" | -+----------------+---------+---------+----------------------------+ -| Through | Yes* | | Misleading when split | -+----------------+---------+---------+----------------------------+ -| Throughout | No | | | -+----------------+---------+---------+----------------------------+ -| **To** | Yes | No | | -+----------------+---------+---------+----------------------------+ -| Toward | No | | | -+----------------+---------+---------+----------------------------+ -| Towards | No | | | -+----------------+---------+---------+----------------------------+ -| Under | No | | | -+----------------+---------+---------+----------------------------+ -| Underneath | No | | | -+----------------+---------+---------+----------------------------+ -| Unlike | No | | | -+----------------+---------+---------+----------------------------+ -| **Until** | Yes | No | | -+----------------+---------+---------+----------------------------+ -| Unto | No | | | -+----------------+---------+---------+----------------------------+ -| Up | No* | | Used as adjective | -+----------------+---------+---------+----------------------------+ -| Upon | Yes* | | Misleading when split | -+----------------+---------+---------+----------------------------+ -| Versus | No | | | -+----------------+---------+---------+----------------------------+ -| **Via** | Yes | No | | -+----------------+---------+---------+----------------------------+ -| **With** | Yes | **Yes** | | -+----------------+---------+---------+----------------------------+ -| **Within** | Yes | No | | -+----------------+---------+---------+----------------------------+ -| Without | Yes* | | Misleading when split | -+----------------+---------+---------+----------------------------+ -| Worth | No | | | -+----------------+---------+---------+----------------------------+ - -.. _the english club: http://www.englishclub.com/grammar/prepositions-list.htm diff --git a/docs/proposals/rejected/Bridging Container Protocols to Class Clusters.md b/docs/proposals/rejected/Bridging Container Protocols to Class Clusters.md new file mode 100644 index 0000000000000..2ed343f765133 --- /dev/null +++ b/docs/proposals/rejected/Bridging Container Protocols to Class Clusters.md @@ -0,0 +1,181 @@ +> **warning** +> +> This proposal was rejected. We ultimately decided to keep Array as + +> a dual-representation struct. + +Bridging Container Protocols to Class Clusters +============================================== + +I think that attempting to bridge `NSArray` to a concrete type like +`Array` will be a poor compromise, losing both the flexibility of +NSArray's polymorphism and the performance afforded by `Array`'s +simplicity. Here's what I propose instead: + +- Rename our current `Array` back to `Vector` or perhaps something + like `ContiguousArray`. +- Redefine `Array` as a refinement of the `Collection` protocol that + has integer indices. +- Implement an `ArrayOf` generic container, like `AnyGenerator` and + `AnySequence`, that can hold an arbitrary type conforming to + `Array`. +- Bridge `NSArray` from ObjC to Swift `ArrayOf` with + value semantics. +- Bridge `Array`-conforming types with class element types in Swift to + ObjC as `NSArray`. + +Although I'll be talking about arrays in this proposal, I think the same +approach would work for `NSDictionary` and `NSSet` as well, mapping them +to generic containers for associative map and and unordered container +protocols respectively. + +NSArray vs Array +---------------- + +Despite their similar names, `NSArray` and Swift's `Array` have +fundamentally incompatible design goals. As the root of a class cluster, +`NSArray` provides abstraction over many underlying data structures, +trading weaker algorithmic guarantees for better representational +flexibility and implementation encapsulation. Swift's `Array`, on the +other hand, is intended to be a direct representation of a contiguous +region of memory, more like a C array or C++'s `vector`, minimizing +abstraction in order to provide tight algorithmic guarantees. Many +`NSArray` implementations are lazy, such as those over KVO properties or +Core Data aggregates, and transforming them to concrete `Array`s would +have unintended semantic effects. And on the other side, the overhead of +having to accommodate an arbitrary `NSArray` implementation inside +`Array` destroys `Array` as a simple, high-performance container. +Attempting to bridge these two types will result in an unattractive +compromise to both sides, weakening the algorithmic guarantees of Array +while forgoing the full flexibility of `NSArray`. + +"Array" as a Refinement of the Collection Protocol +-------------------------------------------------- + +Swift's answer to container polymorphism is its generics system. The +`Collection` protocol provides a common interface to indexable +containers that can be used generically, which is exactly what `NSArray` +provides in Cocoa for integer-indexable container implementations. +`Array` could be described as a refinement of `Collection` with integer +indices: + + protocol Array : Collection { + where IndexType == Int + } + protocol MutableArray : MutableCollection { + where IndexType == Int + } + +The familiar `NSArray` API can then be exposed using default +implementations in the `Array` protocol, or perhaps even on the more +abstract `Collection` and `Sequence` protocols, and we can bridge +`NSArray` in a way that plays nicely with generic containers. + +This naming scheme would of course require us to rename the concrete +`Array` container yet again. `Vector` is an obvious candidate, albeit +one with a C++-ish bent. Something more descriptive like +`ContiguousArray` might feel more Cocoa-ish. + +The ArrayOf<T> Type +------------------------- + +Although the language as implemented does not yet support protocol types +for protocols with associated types, DaveA devised a technique for +implementing types that provide the same effect in the library, such as +his `AnyGenerator` and `AnySequence` containers for arbitrary +`Stream` and `Sequence` types. This technique can be extended to the +`Array` protocol, using class inheritance to hide the concrete +implementing type behind an abstract base: + + // Abstract base class that forwards the Array protocol + class ArrayOfImplBase { + var startIndex: Int { fatal() } + var endIndex: Int { fatal() } + + func __getitem__(i: Int) -> T { fatal() } + + // For COW + func _clone() -> Self { fatal() } + } + + // Concrete derived class containing a specific Array implementation + class ArrayOfImpl + : ArrayOfImplBase + { + var value: ArrayT + var startIndex: Int { return value.startIndex } + var endIndex: Int { return value.endIndex } + func __getitem__(i: Int) -> T { return __getitem__(i) } + + // For COW + func _clone() -> Self { return self(value) } + } + + // Wrapper type that uses the base class to erase the concrete type of + // an Array + struct ArrayOf : Array { + var value: ArrayOfImplBase + + var startIndex: Int { return value.startIndex } + var endIndex: Int { return value.endIndex } + func __getitem__(i: Int) -> T { return value.__getitem__(i) } + + init(arr: ArrayT) { + value = ArrayOfImpl(arr) + } + } + +The mutable variant can use COW optimization to preserve value +semantics: + + struct MutableArrayOf : MutableArray { + /* ...other forwarding methods... */ + + func __setitem__(i: Int, x: T) { + if !isUniquelyReferenced(value) { + value = value._clone() + } + value.__setitem__(i, x) + } + } + +Bridging `NSArray` into Swift +----------------------------- + +We could simply make `NSArray` conform to `Array`, which would be +sufficient to allow it to be stored in an `ArrayOf` +container. However, a good experience for `NSArray` still requires +special-case behavior. In particular, `NSArray` in Cocoa is considered a +value class, and best practice dictates that it be defensively `copy`-ed +when used. In Swift, we should give bridged NSArrays COW value semantics +by default, like `NSString`. One way to handle this is by adding a case +to the `ArrayOf` implementation, allowing it to either contain a generic +value or an `NSArray` with COW semantics. + +Bridging Swift Containers to `NSArray` +-------------------------------------- + +We could have an implicit conversion to `NSArray` from an arbitrary type +conforming to `Array` with a class element type, allowing ObjC APIs to +work naturally with generic Swift containers. Assuming we had support +for `conversion_to` functions, it could look like this: + + class NSArrayOf : NSArray { + /* ...implement NSArray methods... */ + } + + extension NSArray { + @conversion_to + func __conversion_to< + ArrayT: Array where ArrayT.Element : class + >(arr: ArrayT) -> NSArray { + return NSArrayOf(arr) + } + } + +`NSArray` has reference semantics in ObjC, which is a mismatch with +Swift's value semantics, but because `NSArray` is a value class, this is +probably not a problem in practice, because it will be `copy`-ed as +necessary as a best practice. There also needs to be a special case for +bridging an `ArrayOf` that contains an `NSArray`; such a container +should be bridged directly back to the underlying unchanged `NSArray`. diff --git a/docs/proposals/rejected/Bridging Container Protocols to Class Clusters.rst b/docs/proposals/rejected/Bridging Container Protocols to Class Clusters.rst deleted file mode 100644 index cbef36e587f2d..0000000000000 --- a/docs/proposals/rejected/Bridging Container Protocols to Class Clusters.rst +++ /dev/null @@ -1,176 +0,0 @@ -:orphan: - -.. warning:: This proposal was rejected. We ultimately decided to keep Array as - a dual-representation struct. - -Bridging Container Protocols to Class Clusters -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -I think that attempting to bridge ``NSArray`` to a concrete type like -``Array`` will be a poor compromise, losing both the flexibility of NSArray's -polymorphism and the performance afforded by ``Array``'s simplicity. -Here's what I propose instead: - -- Rename our current ``Array`` back to ``Vector`` or perhaps something like - ``ContiguousArray``. -- Redefine ``Array`` as a refinement of the ``Collection`` protocol - that has integer indices. -- Implement an ``ArrayOf`` generic container, like ``AnyGenerator`` and - ``AnySequence``, that can hold an arbitrary type conforming to ``Array``. -- Bridge ``NSArray`` from ObjC to Swift ``ArrayOf`` with value - semantics. -- Bridge ``Array``-conforming types with class element types in Swift to - ObjC as ``NSArray``. - -Although I'll be talking about arrays in this proposal, I think the same -approach would work for ``NSDictionary`` and ``NSSet`` as well, mapping them -to generic containers for associative map and and unordered container protocols -respectively. - -NSArray vs Array -================ - -Despite their similar names, ``NSArray`` and Swift's ``Array`` have -fundamentally incompatible design goals. As the root of a class cluster, -``NSArray`` provides abstraction over many underlying data structures, trading -weaker algorithmic guarantees for better representational flexibility and -implementation encapsulation. Swift's ``Array``, on the other hand, is intended to be a direct -representation of a contiguous region of memory, more like a C array or C++'s -``vector``, minimizing abstraction in order to provide tight algorithmic -guarantees. Many ``NSArray`` implementations are lazy, -such as those over KVO properties or Core Data aggregates, and -transforming them to concrete ``Array``\ s would have unintended semantic -effects. And on the other side, the overhead of having to accommodate an -arbitrary ``NSArray`` implementation inside ``Array`` destroys ``Array`` -as a simple, high-performance container. Attempting to bridge these two types -will result in an unattractive compromise to both sides, weakening the -algorithmic guarantees of Array while forgoing the full flexibility of -``NSArray``. - -"Array" as a Refinement of the Collection Protocol -================================================== - -Swift's answer to container polymorphism is its generics system. The -``Collection`` protocol provides a common interface to indexable containers -that can be used generically, which is exactly what ``NSArray`` provides in -Cocoa for integer-indexable container implementations. ``Array`` could be -described as a refinement of ``Collection`` with integer indices:: - - protocol Array : Collection { - where IndexType == Int - } - protocol MutableArray : MutableCollection { - where IndexType == Int - } - -The familiar ``NSArray`` API can then be exposed using default implementations -in the ``Array`` protocol, or perhaps even on the more abstract ``Collection`` -and ``Sequence`` protocols, and we can bridge ``NSArray`` in a way that plays -nicely with generic containers. - -This naming scheme would of course require us to rename the concrete -``Array`` container yet again. ``Vector`` is an obvious candidate, albeit -one with a C++-ish bent. Something more descriptive like ``ContiguousArray`` -might feel more Cocoa-ish. - -The ArrayOf Type -=================== - -Although the language as implemented does not yet support protocol types for -protocols with associated types, DaveA devised a technique for implementing -types that provide the same effect in the library, such as his ``AnyGenerator`` -and ``AnySequence`` containers for arbitrary ``Stream`` and ``Sequence`` -types. This technique can be extended to the ``Array`` protocol, using class -inheritance to hide the concrete implementing type behind an abstract base:: - - // Abstract base class that forwards the Array protocol - class ArrayOfImplBase { - var startIndex: Int { fatal() } - var endIndex: Int { fatal() } - - func __getitem__(i: Int) -> T { fatal() } - - // For COW - func _clone() -> Self { fatal() } - } - - // Concrete derived class containing a specific Array implementation - class ArrayOfImpl - : ArrayOfImplBase - { - var value: ArrayT - var startIndex: Int { return value.startIndex } - var endIndex: Int { return value.endIndex } - func __getitem__(i: Int) -> T { return __getitem__(i) } - - // For COW - func _clone() -> Self { return self(value) } - } - - // Wrapper type that uses the base class to erase the concrete type of - // an Array - struct ArrayOf : Array { - var value: ArrayOfImplBase - - var startIndex: Int { return value.startIndex } - var endIndex: Int { return value.endIndex } - func __getitem__(i: Int) -> T { return value.__getitem__(i) } - - init(arr: ArrayT) { - value = ArrayOfImpl(arr) - } - } - -The mutable variant can use COW optimization to preserve value semantics:: - - struct MutableArrayOf : MutableArray { - /* ...other forwarding methods... */ - - func __setitem__(i: Int, x: T) { - if !isUniquelyReferenced(value) { - value = value._clone() - } - value.__setitem__(i, x) - } - } - -Bridging ``NSArray`` into Swift -=============================== - -We could simply make ``NSArray`` conform to ``Array``, which would be -sufficient to allow it to be stored in an ``ArrayOf`` container. -However, a good experience for ``NSArray`` still requires special-case -behavior. In particular, ``NSArray`` in Cocoa is considered a value class, -and best practice dictates that it be defensively ``copy``-ed when used. In -Swift, we should give bridged NSArrays COW value semantics by default, like -``NSString``. One way to handle this is by adding a case to the ``ArrayOf`` -implementation, allowing it to either contain a generic value or an ``NSArray`` -with COW semantics. - -Bridging Swift Containers to ``NSArray`` -======================================== - -We could have an implicit conversion to ``NSArray`` from an arbitrary type -conforming to ``Array`` with a class element type, allowing ObjC APIs to work -naturally with generic Swift containers. Assuming we had support for -``conversion_to`` functions, it could look like this:: - - class NSArrayOf : NSArray { - /* ...implement NSArray methods... */ - } - - extension NSArray { - @conversion_to - func __conversion_to< - ArrayT: Array where ArrayT.Element : class - >(arr: ArrayT) -> NSArray { - return NSArrayOf(arr) - } - } - -``NSArray`` has reference semantics in ObjC, which is a mismatch with -Swift's value semantics, but because ``NSArray`` is a value class, this is -probably not a problem in practice, because it will be ``copy``-ed as -necessary as a best practice. There also needs to be a special case for bridging -an ``ArrayOf`` that contains an ``NSArray``; such a container should be -bridged directly back to the underlying unchanged ``NSArray``. diff --git a/docs/proposals/rejected/ClassConstruction.md b/docs/proposals/rejected/ClassConstruction.md new file mode 100644 index 0000000000000..1137ed861bac7 --- /dev/null +++ b/docs/proposals/rejected/ClassConstruction.md @@ -0,0 +1,126 @@ +Integrating Swift Constructors with Objective-C +=============================================== + +> **warning** +> +> This proposal was rejected, though it helped in the design of the + +> final Swift 1 initialization model. + +Objective-C's “designated inititalizers” pattern seems at first to +create a great deal of complication. However, designated initializers +are simply the only sane response to Objective-C's initialization rules, +which are the root cause of the complication. + +This proposal suggests an approach to initialization that avoids the +problems inherent in Objective-C while still *allowing* Objective-C +programmers to pursue the designated initializer pattern on subclasses +of Swift classes. + +The Root of the Problem +----------------------- + +The root problem with Objective-C's initialization rules is that the +`init` methods of a superclass automatically become public members of +its subclasses. This leads to a soundness problem: + +Because there is no way to hide a superclass' `init` method from +clients, ensuring that subclass instances are properly initialized +requires overriding *every* superclass initializer in *every* subclass: + +Following this rule is obviously tedious and error-prone for users. +Initialization is crucial to correctness, because it is where invariants +are established. It therefore should be no more complex than everything +else to reason about. + +Also, it means adding an `init` method in a base class can be +API-breaking. + +Furthermore, as John McCall pointed out recently, it forces +inappropriate interfaces on subclasses. For example, every subclass of +`NSObject` has a parameter-less `init` function, whether or not there's +an appropriate way to construct instances of that subclass without +parameters. As a result, class designers may be forced to expose weaker +invariants than the ones they could otherwise establish. + +Exceptions to the Rule +---------------------- + +I exaggerated a little in the previous section: because overriding +*every* superclass initializer in *every* subclass is so tedious, the +Objective C community has identified some situations where you don't +really need to override every `init` method: + +1. When you know the default zero-initialization of a class' instance + variables is good enough, you don't need to override any `init` + methods from your superclass. +2. If a given superclass' `init` method always calls another `init` + method, you don't need to override the first `init` method because + your instance variables will be initialized by your override of the + second `init` method. In this case, the first (outer) `init` method + is called a **secondary initializer**. Any `init` method that's not + secondary is called a **designated initializer**. + +How to Think About This +----------------------- + +At this point I'll make a few assertions that I hope will be +self-evident, given the foregoing context: + +1. If the programmer follows all the rules correctly, one initializer + is as good as another: every `init` method, whether designated or + secondary, fully initializes all the instance variables. This is + true for all clients of the class, including subclassers. +2. Distinguishing designated from secondary initializers does nothing + to provide soundness. It's *merely* a technique for limiting the + tedious `init` method overrides required of users. +3. Swift users would not be well-served by a construction model that + exposes superclass `init` methods to clients of subclasses + by default. + +Proposal +-------- + +I suggest we define Swift initialization to be as simple and +easily-understood as possible, and avoid “interesting” interactions with +the more complicated Objective-C initialization process. If we do this, +we can treat Objective-C base classes as “sealed and safe” for the +purpose of initialization, and help programmers reason effectively about +initialization and their class invariants. + +Here are the proposed rules: + +- `init` methods of base classes defined in Objective-C are not, by + default, part of the public interface of a subclass defined + in Swift. +- `init` methods of base classes defined in Swift are not, by default, + part of the public interface of a subclass defined in Objective-C. +- `self.init(…)` calls in Swift never dispatch virtually. We have a + safe model for “virtual initialization:” `init` methods can call + overridable methods after all instance variables and superclasses + are initialized. Allowing *virtual* constructor delegation would + undermine that safety. +- As a convenience, when a subclass' instance variables all have + initializers, it should be possible to explicitly expose superclass + init methods in a Swift subclass without writing out complete + forwarding functions. For example: + + @inherit init(x:y:z) // one possible syntax + + > **note** + > + > Allowing `@inherit init(*)` is a terrible idea + > + > It allows superclasses to break their subclasses by adding + > `init` methods. + +Summary +------- + +By eliminating by-default `init`method inheritance and disabling virtual +dispatch in constructor delegation, we give class designers full control +over the state of their constructed instances. By preserving virtual +dispatch for non-`self`, non-`super` calls to `init` methods, we allow +Objective-C programmers to keep using the patterns that depend on +virtual dispatch, including designated initializers and `initWithCoder` +methods. diff --git a/docs/proposals/rejected/ClassConstruction.rst b/docs/proposals/rejected/ClassConstruction.rst deleted file mode 100644 index 9db9f65ead8a8..0000000000000 --- a/docs/proposals/rejected/ClassConstruction.rst +++ /dev/null @@ -1,163 +0,0 @@ -:orphan: - -================================================= - Integrating Swift Constructors with Objective-C -================================================= - -.. warning:: This proposal was rejected, though it helped in the design of the - final Swift 1 initialization model. - -Objective-C's “designated inititalizers” pattern seems at first to -create a great deal of complication. However, designated initializers -are simply the only sane response to Objective-C's initialization rules, -which are the root cause of the complication. - -This proposal suggests an approach to initialization that avoids the -problems inherent in Objective-C while still *allowing* Objective-C -programmers to pursue the designated initializer pattern on subclasses -of Swift classes. - -The Root of the Problem -======================= - -The root problem with Objective-C's initialization rules is that the -``init`` methods of a superclass automatically become public members -of its subclasses. This leads to a soundness problem: - -.. parsed-literal:: - - @interface SuperClass - - initSuperClass - @end - - @interface Subclass : Superclass - - (void)subclassMethod - @end - - @implementation Subclass : Superclass - char\* **name**\ ; // never initialized - - - (void)print { printf(\ **name**\ ); } // oops - @end - - mySubclassInstance = [ [Subclass alloc] initSuperClass ] - -Because there is no way to hide a superclass' ``init`` method from -clients, ensuring that subclass instances are properly initialized -requires overriding *every* superclass initializer in *every* -subclass: - -.. parsed-literal:: - - @implementation Subclass : Superclass - char\* name; - - initSuperClass { - [super initSuperClass]; // Don't forget the superclass - **name = "Tino";** - } - - (void)print { printf(name); } // OK - @end - -Following this rule is obviously tedious and error-prone for users. -Initialization is crucial to correctness, because it is where -invariants are established. It therefore should be no more complex -than everything else to reason about. - -Also, it means adding an ``init`` method in a base class can be -API-breaking. - -Furthermore, as John McCall pointed out recently, it forces -inappropriate interfaces on subclasses. For example, every subclass -of ``NSObject`` has a parameter-less ``init`` function, whether or not -there's an appropriate way to construct instances of that subclass -without parameters. As a result, class designers may be forced to -expose weaker invariants than the ones they could otherwise establish. - -Exceptions to the Rule -====================== - -I exaggerated a little in the previous section: because overriding -*every* superclass initializer in *every* subclass is so tedious, the -Objective C community has identified some situations where you don't -really need to override every ``init`` method: - -1. When you know the default zero-initialization of a class' instance - variables is good enough, you don't need to override any ``init`` - methods from your superclass. - -2. If a given superclass' ``init`` method always calls another - ``init`` method, you don't need to override the first ``init`` - method because your instance variables will be initialized by your - override of the second ``init`` method. In this case, the first - (outer) ``init`` method is called a **secondary initializer**. Any - ``init`` method that's not secondary is called a **designated - initializer**. - -How to Think About This -======================= - -At this point I'll make a few assertions that I hope will be -self-evident, given the foregoing context: - -1. If the programmer follows all the rules correctly, one initializer - is as good as another: every ``init`` method, whether designated or - secondary, fully initializes all the instance variables. This is - true for all clients of the class, including subclassers. - -2. Distinguishing designated from secondary initializers does nothing - to provide soundness. It's *merely* a technique for limiting the - tedious ``init`` method overrides required of users. - -3. Swift users would not be well-served by a construction model that - exposes superclass ``init`` methods to clients of subclasses by - default. - -Proposal -======== - -I suggest we define Swift initialization to be as simple and -easily-understood as possible, and avoid “interesting” interactions -with the more complicated Objective-C initialization process. If we -do this, we can treat Objective-C base classes as “sealed and safe” -for the purpose of initialization, and help programmers reason -effectively about initialization and their class invariants. - -Here are the proposed rules: - -* ``init`` methods of base classes defined in Objective-C are not, by - default, part of the public interface of a subclass defined in - Swift. - -* ``init`` methods of base classes defined in Swift are not, by - default, part of the public interface of a subclass defined in - Objective-C. - -* ``self.init(…)`` calls in Swift never dispatch virtually. We have a - safe model for “virtual initialization:” ``init`` methods can call - overridable methods after all instance variables and superclasses - are initialized. Allowing *virtual* constructor delegation would - undermine that safety. - -* As a convenience, when a subclass' instance variables all have - initializers, it should be possible to explicitly expose superclass - init methods in a Swift subclass without writing out complete - forwarding functions. For example:: - - @inherit init(x:y:z) // one possible syntax - - .. Note:: Allowing ``@inherit init(*)`` is a terrible idea - - It allows superclasses to break their subclasses by adding - ``init`` methods. - - -Summary -======= - -By eliminating by-default ``init``\ method inheritance and disabling -virtual dispatch in constructor delegation, we give class designers -full control over the state of their constructed instances. By -preserving virtual dispatch for non-``self``, non-``super`` calls to -``init`` methods, we allow Objective-C programmers to keep using the -patterns that depend on virtual dispatch, including designated -initializers and ``initWithCoder`` methods. diff --git a/docs/proposals/rejected/Clonable.md b/docs/proposals/rejected/Clonable.md new file mode 100644 index 0000000000000..cd2849ccd518a --- /dev/null +++ b/docs/proposals/rejected/Clonable.md @@ -0,0 +1,156 @@ +Clonable +======== + +Author + +: Dave Abrahams + +Author + +: Joe Groff + +Date + +: 2013-03-21 + +Edition + +: 2 + +> **warning** +> +> This proposal was rejected. We decided not to introduce a + +> language-level copying mechanism for classes. + +**Abstract:** to better support the creation of value types, we propose +a “magic” `Clonable` protocol and an annotation for describing which +instance variables should be cloned when a type is copied. This proposal +**augments revision 1** of the Clonable proposal with our rationale for +dropping our support for `val` and `ref`, a description of the +programming model for generics, and a brief discussion of equality. It +is **otherwise unchanged**. + +Rationale +--------- + +By eliminating `val`, we lose the easy creation of runtime-polymorphic +value types. Instead of merely writing: + + val x : MyClass + +one has to engage in some kind of wrapping and forwarding: + + struct MyClassVal { + var [clone] value : MyClass + + constructor(x : A, y : B) { + value = new MyClass(x, y) + } + + func someFunction(z : C) -> D { + return value.someFunction(z) + } + + // ...etc... + } + +Although such wrapping is awful, this is not the only place where one +would want to do it. Therefore, some kind of ability to forward an +entire interface wholesale could be added as a separate extension +(getter/setter for `This`?), which would solve more problems. Then it +would be easy enough to write the wrapper as a generic struct and +`Val` would be a reality. + +By eliminating `ref`, we lose the easy creation of references to value +types. However, among those who prefer programming with values, having +an explicit step for dereferencing might make more sense, so we could +use this generic class: + +> class Reference<T> { value : T } + +If explicit dereferencing isn't desired, there's always manual (or +automatic, if we add that feature) forwarding. + +By dropping `val` we also lose some terseness aggregating `class` +contents into `struct`s. However, since `ref` is being dropped there's +less call for a symmetric `val`. The extra “cruft” that `[clone]` adds +actually seems appropriate when viewed as a special bridge for `class` +types, and less like a penalty against value types. + +Generics +-------- + +There is actually a straightforward programming model for generics. If +you want to design a generic component where a type parameter `T` binds +to both `class`es and non-`class` types, you can view `T` as a value +type where—as with C pointers—the value is the reference rather than the +object being referred to. + +Of course, if `T` is only supposed to bind to `class`es, a different +programming model may work just as well. + +Implications for Equality +------------------------- + +We think the programming model suggested for generics has some pretty +strong implications for equality of `class`es: `a == b` must return true +iff `a` and `b` refer to the same object. + +Details *(unchanged from Revision 1)* +------------------------------------- + +When a type with reference semantics `R` is to be used as a part of +(versus merely being referred-to-by) a type with value semantics `V`, a +new annotation, `[clone]` can be used to indicate that the `R` instance +variable should be `clone()`d when `V` is copied. + +A `class` can be `clone()`d when it implements the built-in `Clonable` +protocol: + + protocol Clonable { + func clone() -> Self { /* see below */ } + } + +The implementation of `clone()` (which will be generated by the compiler +and typically never overridden) performs a primitive copy of all +ordinary instance variables, and a `clone()` of all instance variables +marked `[clone]`: + + class FooValue : Clonable {} + + class Bar {} + + class Foo : Clonable { + var count : Int + var [clone] myValue : FooValue + var somethingIJustReferTo : Bar + } + + struct Baz { + var [clone] partOfMyValue : Foo + var anotherPart : Int + var somethingIJustReferTo : Bar + } + +When a `Baz` is copied by any of the “big three” operations (variable +initialization, assignment, or function argument passing), even as part +of a larger `struct`, its `[clone]` member is `clone()`d. Because `Foo` +itself has a `[clone]` member, that is `clone()`d also. Therefore +copying a `Baz` object `clone()`s a `Foo` and `clone()`ing a `Foo` +`clone()`s a `FooValue`. + +All `struct`s are `Clonable` by default, with `clone()` delivering +ordinary copy semantics. Therefore, : + + var x : Baz + var y = x.clone() + +is equivalent to : + + var x : Baz + var y = x + +Note that `Clonable` is the first protocol with a default implementation +that can't currently be written in the standard library (though arguably +we'd like to add the capability to write that implementation). diff --git a/docs/proposals/rejected/Clonable.rst b/docs/proposals/rejected/Clonable.rst deleted file mode 100644 index 949ecf93cb883..0000000000000 --- a/docs/proposals/rejected/Clonable.rst +++ /dev/null @@ -1,150 +0,0 @@ -:orphan: - -.. @raise litre.TestsAreMissing - -========== - Clonable -========== - -:Author: Dave Abrahams -:Author: Joe Groff -:Date: 2013-03-21 -:Edition: 2 - -.. warning:: This proposal was rejected. We decided not to introduce a - language-level copying mechanism for classes. - -**Abstract:** to better support the creation of value types, we -propose a “magic” ``Clonable`` protocol and an annotation for describing -which instance variables should be cloned when a type is copied. This -proposal **augments revision 1** of the Clonable proposal with our -rationale for dropping our support for ``val`` and ``ref``, a -description of the programming model for generics, and a brief -discussion of equality. It is **otherwise unchanged**. - -Rationale -========= - -By eliminating ``val``, we lose the easy creation of -runtime-polymorphic value types. Instead of merely writing:: - - val x : MyClass - -one has to engage in some kind of wrapping and forwarding:: - - struct MyClassVal { - var [clone] value : MyClass - - constructor(x : A, y : B) { - value = new MyClass(x, y) - } - - func someFunction(z : C) -> D { - return value.someFunction(z) - } - - // ...etc... - } - -Although such wrapping is awful, this is not the only place where one -would want to do it. Therefore, some kind of ability to forward an -entire interface wholesale could be added as a separate extension -(getter/setter for ``This``?), which would solve more problems. Then it -would be easy enough to write the wrapper as a generic struct and -``Val`` would be a reality. - -By eliminating ``ref``, we lose the easy creation of references to -value types. However, among those who prefer programming with values, -having an explicit step for dereferencing might make more sense, so we -could use this generic class: - - class Reference { value : T } - -If explicit dereferencing isn't desired, there's always manual (or -automatic, if we add that feature) forwarding. - -By dropping ``val`` we also lose some terseness aggregating ``class`` -contents into ``struct``\ s. However, since ``ref`` is being dropped -there's less call for a symmetric ``val``. The extra “cruft” that -``[clone]`` adds actually seems appropriate when viewed as a special -bridge for ``class`` types, and less like a penalty against value -types. - -Generics -======== - -There is actually a straightforward programming model for generics. -If you want to design a generic component where a type parameter ``T`` -binds to both ``class``\ es and non-``class`` types, you can view -``T`` as a value type where—as with C pointers—the value is the -reference rather than the object being referred to. - -Of course, if ``T`` is only supposed to bind to ``class``\ es, a -different programming model may work just as well. - -Implications for Equality -========================= - -We think the programming model suggested for generics has some pretty -strong implications for equality of ``class``\ es: ``a == b`` must -return true iff ``a`` and ``b`` refer to the same object. - -Details *(unchanged from Revision 1)* -===================================== - -When a type with reference semantics ``R`` is to be used as a part of -(versus merely being referred-to-by) a type with value semantics ``V``, -a new annotation, ``[clone]`` can be used to indicate that the ``R`` -instance variable should be ``clone()``\ d when ``V`` is copied. - -A ``class`` can be ``clone()``\ d when it implements the built-in ``Clonable`` -protocol:: - - protocol Clonable { - func clone() -> Self { /* see below */ } - } - -The implementation of ``clone()`` (which will be generated by the -compiler and typically never overridden) performs a primitive copy of -all ordinary instance variables, and a ``clone()`` of all instance -variables marked ``[clone]``:: - - class FooValue : Clonable {} - - class Bar {} - - class Foo : Clonable { - var count : Int - var [clone] myValue : FooValue - var somethingIJustReferTo : Bar - } - - struct Baz { - var [clone] partOfMyValue : Foo - var anotherPart : Int - var somethingIJustReferTo : Bar - } - -When a ``Baz`` is copied by any of the “big three” operations (variable -initialization, assignment, or function argument passing), even as -part of a larger ``struct``, its ``[clone]`` member is ``clone()``\ d. -Because ``Foo`` itself has a ``[clone]`` member, that is ``clone()``\ d -also. Therefore copying a ``Baz`` object ``clone()``\ s a ``Foo`` and -``clone()``\ ing a ``Foo`` ``clone()``\ s a ``FooValue``. - -All ``struct``\ s are ``Clonable`` by default, with ``clone()`` delivering -ordinary copy semantics. Therefore, :: - - var x : Baz - var y = x.clone() - -is equivalent to :: - - var x : Baz - var y = x - -Note that ``Clonable`` is the first protocol with a default -implementation that can't currently be written in the standard library -(though arguably we'd like to add the capability to write that -implementation). - diff --git a/docs/proposals/rejected/Constructors.md b/docs/proposals/rejected/Constructors.md new file mode 100644 index 0000000000000..ea5d4bd988ebe --- /dev/null +++ b/docs/proposals/rejected/Constructors.md @@ -0,0 +1,555 @@ +orphan + +: + +Constructors and Initialization in Swift +======================================== + +> **warning** +> +> This proposal was rejected, though it helped in the design of the + +> final Swift 1 initialization model. + +Initialization in Objective-C +----------------------------- + +In Objective-C, object allocation and initialization are separate +operations that are (by convention) always used together. One sends the +`alloc` message to the class to allocate it, then sends an `init` (or +other message in the init family) to the newly-allocated object, for +example: + + [[NSString alloc] initWithUTF8String:"initialization"] + +### Two-Phase Initialization + +The separation of allocation and initialization implies that +initialization is a two-phase process. In the first phase (allocation), +memory is allocated and the memory associated with all instance +variables is zero'd out. In the second phase, in which the appropriate +"init" instance method is invoked, the instance variables are (manually) +initialized. + +Note that an object is considered to be fully constructor after +allocation but before initialization. Therefore, a message send +initiated from a superclass's `init` can invoke a method in a subclass +whose own `init` has not completed. A contrived example: + + @interface A : NSObject { + NSString *_description; + } + - (id)init; + - (NSString*)description; + @end + + @implementation A + - (id)init { + self = [super init] + if (self) { + _description = [self description]; + } + return self; + } + + - (NSString *)description { + return @"A"; + } + @end + + @interface B : A + @property NSString *title; + @end + + @implementation B + - (id)init { + self = [super init] + if (self) { + self->title = @"Hello"; + } + return self; + } + + -(NSString *)description { + return self->title; + } + @end + +During the second phase of initialization, A's `-init` method invokes +the `-description` method, which ends up in B's `-description`. Here, +`title` will be `nil` even though the apparent invariant (when looking +at B's `-init` method) is that `title` is never nil, because B's `-init` +method has not yet finished execution. + +In a language with single-phase initialization (such as C++, Java, or +C\# constructors), the object is considered to have the type of the +constructor currently executing for the purposes of dynamic dispatch. +For the Objective-C example above, this would mean that when A's `-init` +sends the `description` message, it would invoke A's `-description`. +This is somewhat safer than two-phase initialization, because the +programmer does not have to deal with the possibility of executing one's +methods before the initialization of one's instance variables have +completed. It is also less flexible. + +### Designated Initializers + +One of the benefits of Objective-C's `init` methods is that they are +instance methods, and therefore are inherited. One need not override the +`init` methods in a subclass unless that subclass has additional +instance variables that require initialization. However, when a subclass +does introduce additional instance variables that require +initialization, which `init` methods should it override? If the answer +is "all of them", then initializer inheritance isn't all that useful. + +Objective-C convention has the notion of a [designated +initializer](https://developer.apple.com/library/ios/documentation/general/conceptual/CocoaEncyclopedia/Initialization/Initialization.html) +which is, roughly, an initializer method that is responsible for calling +one of it's superclass's initializers, then initializing its own +instance variables. Initializers that are not designated initializers +are "secondary" initializers: they typically delegate to another +initializer (eventually terminating the chain at a designated +initializer) rather than performing initialization themselves. For +example, consider the following `Task` class (from the aforementioned +"Concepts in Objective-C Programming" document): + + @interface Task + @property NSString *title; + @property NSDate *date; + + - (id)initWithTitle:(NSString *)aTitle date:(NSDate *)aDate; + - (id)initWithTitle:(NSString *)aTitle; + - (id)init; + @end + + @implementation Task + - (id)initWithTitle:(NSString *)aTitle date:(NSDate *)aDate { + title = aTitle; + date = aDate; + return self; + } + + - (id)initWithTitle:(NSString *)aTitle { + return [self initWithTitle:aTitle date:[NSDate date]]; + } + + - (id)init { + return [self initWithTitle:@"Task"]; + } + @end + +The first initializer is the designated initializer, which directly +initializes the instance variables from its parameters. The second two +initializers are secondary initializers, which delegate to other +initializers, eventually reaching the designated initializer. + +A subclass should override all of its superclass's designated +initializers, but it need not override the secondary initializers. We +can illustrate this with a subclass of `Task` that introduces a new +instance variable: + + @interface PackagedTask : Task + @property dispatch_queue_t queue; + + - (id)initWithTitle:(NSString *)aTitle date:(NSDate *)aDate; + - (id)initWithTitle:(NSString *)aTitle date:(NSDate *)aDate queue:(dispatch_queue_t)aQueue; + @end + + @implementation PackagedTask + - (id)initWithTitle:(NSString *)aTitle date:(NSDate *)aDate { + return [self initWithTitle:aTitle + date:aDate + queue:dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)]; + } + + - (id)initWithTitle:(NSString * aTitle date:(NSDate *)aDate queue:(dispatch_queue_t)aQueue { + self = [super initWithTitle:aTitle date:aDate]; + if (self) { + queue = aQueue; + } + return self; + } + @end + +`PackagedTask` overrides `Task`'s designated initializer, +`-initWithTitle:date:`, which then becomes a secondary initializer of +`PackagedTask`, whose only designated initializer is +`initWithTitle:date:queue:`. The latter method invokes its superclass's +designated initializer (`-[Task initWithTitle:date:]`), then initializes +its own instance variable. By following the rules of designated +initializers mentioned above, one ensures that the inherited secondary +initializers still work. Consider the execution of +`[[PackagedTask alloc] init]`: + +- A `PackagedTask` object is allocated. +- `-[Task init]` executes, which delegates to `-[Task initWithTitle]`. +- `-[Task initWithTitle]` delegates to `-initWithTitle:date:`, which + executes `-[PackagedTask initWithTitle:date:]`. +- `-[PackagedTask initWithTitle:date:]` invokes + `-[Task initWithTitle:date:]` to initialize `Task`'s instance + variables, then initializes its own instance variables. + +### The Middle Phase of Initialization + +Objective-C's two-phase initialization actually has a third part, which +occurs between the zeroing of the instance variables and the call to the +initialization. This initialization invokes the default constructors for +instance variables of C++ class type when those default constructors are +not trivial. For example: + + @interface A + struct X { + X() { printf("X()\n"); } + }; + + @interface A : NSObject { + X x; + } + @end + +When constructing an object with `[[A alloc] init]`, the default +constructor for `X` will execute after the instance variables are zeroed +but before `+alloc` returns. + +Swift Constructors +------------------ + +Swift's constructors merge both allocation and initialization into a +single function. One constructs a new object with type construction +syntax as follows: + + Task(title:"My task", date:NSDate()) + +The object will be allocated (via Swift's allocation routines) and the +corresponding constructor will be invoked to perform the initialization. +The constructor itself might look like this: + + class Task { + var title : String + var date : NSDate + + constructor(title : String = "Task", date : NSDate = NSDate()) { + self.title = title + self.date = date + } + } + +Due to the use of default arguments, one can create a task an any of the +following ways: + + Task() + Task(title:"My task") + Task(date:NSDate()) + Task(title:"My task", date:NSDate()) + +### Constructor Delegation + +Although our use of default arguments has eliminated the need for the +various secondary constructors in the Objective-C version of the `Task` +class, there are other reasons why one might want to have one +constructor in a class call another to perform initialization (called +constructor *delegation*). For example, default arguments cannot make +use of other argument values, and one may have a more complicated +default value. For example, the default title could depend on the +provided date. Swift should support constructor delegation for this use +case: + + constructor(title : String, date : NSDate = NSDate()) { + self.title = title + self.date = date + } + + constructor(date : NSDate = NSDate()) { + /*self.*/constructor(title:"Task created on " + date.description(), + date:date) + } + +A constructor that delegates to another constructor must do so before +using or initializing any of its instance variables. This property can +be verified via definite initialization analysis. + +### Superclass Constructors + +When one class inherits another, each constructor within the subclass +must call one of its superclass's constructors before using or +initializing any of its instance variables, which is also verified via +definite initialization analysis. For example, the Swift `PackagedTask` +class could be implemented as: + + class PackagedTask : Task { + var queue : dispatch_queue_t + + constructor(title : String, date : NSDate = NSDate(), + queue : dispatch_queue_t = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)) { + super.constructor(title:title, date:date) + self.queue = queue + } + } + +### Instance Variable Initialization + +Swift allows instance variables to be provided with an initializer, +which will be called as part of initialization of an object. For +example, we could provide initializers for the members of `Task` as +follows: + + class Task { + var title : String = "Title" + var date : NSDate = NSDate() + } + +Here, one does not need to write any constructor: a default, +zero-parameter constructor will be synthesized by the compiler, which +runs the provided instance variable initializations. Similarly, if I did +write a constructor but did not initialize all of the instance +variables, each uninitialized instance variable would be initialized by +its provided initializer. While this is mainly programmer convenience +(one need only write the common initialization for an instance variable +once, rather than once per non-delegating constructor), it may also have +an impact on the overall initialization story (see below). + +### One- or Two-Phase Initialization? + +Swift's current model it attempts to ensure that instance variables are +initialized before they are used via definite initialization analysis. +However, we haven't yet specified whether dynamic dispatch during +initialization sees the object as being of the currently-executing +constructor's type (as in C++/Java/C\#) or of the final subclass's type +(as in Objective-C). + +I propose that we follow the C++/Java/C\# precedent, which allows us to +ensure (within Swift code) that an instance variable is never accessed +before it has been initialized, eliminating the safety concerns +introduced by Objective-C's two-phase initialization. Practically +speaking, this means that the vtable/isa pointer should be set to the +constructor's type while the constructor executes. + +This model complicates constructor inheritance considerably. A secondary +initializer in Objective-C works by delegating to (eventually) a +designated initializer, which is overridden by the subclass. Following +the C++/Java/C\# precedent breaks this pattern, because the overriding +designated initializer will never be invoked. + +### Constructor Inheritance + +Currently, constructors in Swift are not inherited. This is a limitation +that Swift's constructor model shares with C++98/03, Java, and C\#, and +a regression from Objective-C. C++11 introduced the notion of inherited +constructors. It is an opt-in feature, introduced into one's class with +a using declaration such as: + + using MySuperclass::MySuperclass; + +C++11 inherited constructors are implemented by essentially copying the +signatures of all of the superclass's constructors into the subclass, +ignoring those for which the subclass already has a constructor with the +same signature. This approach does not translate well to Swift, because +there is no way to gather all of the constructors in either the +superclass or the subclass due to the presence of class extensions. + +One potential approach is to bring Objective-C's notion of designated +and secondary initializers into Swift. A "designated" constructor is +responsible for calling the superclass constructor and then initializing +its own instance variables. + +A "secondary" constructor can be written in the class definition or an +extension. A secondary constructor must delegate to another constructor, +which will find the constructor to delegate to based on the type of the +eventual subclass of the object. Thus, the subclass's override of the +designated constructor will initialize all of the instance variables +before continuing execution of the secondary constructor. + +For the above to be safe, we need to ensure that subclasses override all +of the designated constructors of their superclass. Therefore, we +require that designated constructors be written within the class +definition [^1]. Secondary constructors can be written in either the +class definition or an extension. + +In Objective-C, classes generally only have one or two designated +initializers, so having to override them doesn't seem too onerous. If it +begins to feel like boilerplate, we could perform the override +automatically when all of the instance variables of the subclass +themselves have initializers. + +Note that the requirement that all designated constructors be +initializable means that adding a new designated constructor to a class +is both an ABI- and API-breaking change. + +Syntactically, we could introduce some kind of attribute or keyword to +distinguish designated constructors from secondary constructors. Straw +men: `[designated]` for designated constructors, which is implied for +constructors in the class definition, and `[inherited]` for secondary +constructors, which is implied for constructors in class extensions. + +### Class Clusters and Assignment to Self + +TBD. + +Objective-C Interoperability +---------------------------- + +The proposed Swift model for constructors needs to interoperate well +with Objective-C, both for Objective-C classes imported into Swift and +for Swift classes imported into Objective-C. + +### Constructor <-> Initializer Mapping + +Fundamental to the interoperability story is that Swift constructors +serve the same role as Objective-C initializers, and therefore should be +interoperable. An Objective-C object should be constructible with Swift +construction syntax, e.g.,: + + NSDate() + +and one should be able to override an Objective-C initializer in a Swift +subclass by writing a constructor, e.g.,: + + class Task : NSObject { + constructor () { + super.constructor() // invokes -[NSObject init] + + // perform initialization + } + } + +On the Objective-C side, one should be able to create a `Task` object +with `[[Task alloc] init]`, subclass `Task` to override `-init`, and so +on. + +Each Swift constructor has both its two Swift entry points (one that +allocates the object, one that initializes it) and an Objective-C entry +point for the initializer. Given a Swift constructor, the selector for +the Objective-C entry point is formed by: + +- For the first selector piece, prepending the string ''init'' to the + capitalized name of the first parameter. +- For the remaining selector pieces, the names of the + remaining parameters. + +For example, given the Swift constructor: + + constructor withTitle(aTitle : String) date(aDate : NSDate) { + // ... + } + +the Objective-C entry point will have the selector +`initWithTitle:date:`. Missing parameter names will be holes in the +selector (e.g., ''init::::''). If the constructor either has zero +parameters or if it has a single parameter with `Void` type (i.e., the +empty tuple `()`), the selector will be a zero-argument selector with no +trailing colon, e.g.,: + + constructor toMemory(_ : ()) { /* ... */ } + +maps to the selector `initToMemory`. + +This mapping is reversible: given a selector in the “init” family, i.e., +where the first word is “init”, we split the selector into its various +pieces at the colons: + +- For the first piece, we remove the “init” and then lowercase the + next character *unless* the second character is also uppercase. This + becomes the name of the first parameter to the constructor. If this + string is non-empty and the selector is a zero-argument selector + (i.e., no trailing `:`), the first (only) parameter has `Void` type. +- For the remaining pieces, we use the selector piece as the name of + the corresponding parameter to the constructor. + +Note that this scheme intentionally ignores methods that don't have the +leading `init` keyword, including (e.g.) methods starting with `_init` +or methods with completely different names that have been tagged as +being in the `init` family in Objective-C (via the `objc_method_family` +attribute in Clang). We consider these cases to be rare enough that we +don't want to pessimize the conventional `init` methods to accommodate +them. + +### Designated Initializers + +Designed initializers in Objective-C are currently identified by +documentation and convention. To strengthen these conventions and make +it possible to mechanically verify (in Swift) that all designated +initializers have been overridden, we should introduce a Clang method +attribute `objc_designated_initializer` that can be applied to the +designated initializers in Objective-C. Clang can then be extended to +perform similar checking to what we're describing for Swift: designated +initializers delegate or chain to the superclass constructor, secondary +constructors always delegate, and subclassing rrequires one to override +the designated initializers. The impact can be softened somewhat using +warnings or other heuristics, to be (separately) determined. + +### Nil and Re-assigned Self + +An Objective-C initializer can return a self pointer that is different +than the one it was called with. When this happens, it is either due to +an error (in which case it will return nil) or because the object is +being substituted for another object. + +In both cases, we are left with a partially-constructed object that then +needs to be destroyed, even though its instance variables may not yet +havee been initialized. This is also a problem for Objective-C, which +makes returning anything other than the original ''self'' brittle. + +In Swift, we will have a separate error-handling mechanism to report +failures. A Swift constructor will not be allowed to return a value; +rather, it should raise an error if an error occurs, and that error will +be propagated however we eventually decide to implement error +propagation. + +Object substitution is the more complicated feature. I propose that we +do not initially support object substitution within Swift constructors. +However, for memory safety we do need to recognize when calling the +superclass constructor or delegating to another constructor has replaced +the object (which can happen in Objective-C code). Aside from the need +to destroy the original object, the instance variables of the new object +will already have been initialized, so our "initialization" of those +instance variables is actually assignment. The code generation for +constructors will need to account for this. + +Alternatives +------------ + +This proposal is complicated, in part because it's trying to balance the +safety goals of Swift against the convience of Objective-C's two-phase +initialization. + +### Separate Swift Constructors from Objective-C Initializers + +Rather than try to adapt Swift's constructors to work with Objective-C's +initializers, we could instead keep these features distinct. Swift +constructors would maintain their current behavior (which ensures that +instance variables are always initialized), and neither override nor +introduce Objective-C initializers as entry points. + +In this world, one would still have to implement `init` methods in Swift +classes to override Objective-C initializers or make a Swift object +constructible in Objective-C via the `alloc/init` pattern. Swift +constructors would not be inherited, or would use some mechanism like +C++11's inherited constructors. As we do today, Swift's Clang importer +could introduce constructors into Objective-C classes that simply +forward to the underlying initializers, so that we get Swift's object +construction syntax. However, the need to write `init` methods when +subclasses would feel like a kludge. + +### Embrace Two-Phase Initialization + +We could switch Swift whole-heartedly over to two-phase initialization, +eliminating constructors in favor of `init` methods. We would likely +want to ensure that all instance variables get initialized before the +`init` methods ever run, for safety reasons. This could be handled by +(for example) requiring initializers on all instance variables, and +running those initializers after allocation and before the `init` +methods are called, the same way that instance variables with +non-trivial default constructors are handled in Objective-C++. We would +still likely need the notion of a designated initializer in Objective-C +to make this safe in Swift, since we need to know which Objective-C +initializers are guaranteed to initialize those instance variables not +written in Swift. + +This choice makes interoperability with Objective-C easier (since we're +adopting Objective-C's model), but it makes safety either harder (e.g., +we have to make all of our methods guard against uninitialized instance +variables) or more onerous (requiring initializers on the declarations +of all instance variables). + +[^1]: We could be slightly more lenient here, allowing designated + constructors to be written in any class extension within the same + module as the class definition. diff --git a/docs/proposals/rejected/Constructors.rst b/docs/proposals/rejected/Constructors.rst deleted file mode 100644 index 69fcb9660217f..0000000000000 --- a/docs/proposals/rejected/Constructors.rst +++ /dev/null @@ -1,584 +0,0 @@ -:orphan: - -Constructors and Initialization in Swift -======================================== - -.. warning:: This proposal was rejected, though it helped in the design of the - final Swift 1 initialization model. - -.. contents:: - -Initialization in Objective-C ------------------------------ - -In Objective-C, object allocation and initialization are separate -operations that are (by convention) always used together. One sends -the ``alloc`` message to the class to allocate it, then sends an -``init`` (or other message in the init family) to the newly-allocated -object, for example:: - - [[NSString alloc] initWithUTF8String:"initialization"] - -Two-Phase Initialization -~~~~~~~~~~~~~~~~~~~~~~~~ - -The separation of allocation and initialization implies that -initialization is a two-phase process. In the first phase -(allocation), memory is allocated and the memory associated with all -instance variables is zero'd out. In the second phase, in which the -appropriate "init" instance method is invoked, the instance variables -are (manually) initialized. - -Note that an object is considered to be fully constructor after -allocation but before initialization. Therefore, a message send -initiated from a superclass's ``init`` can invoke a method in a -subclass whose own ``init`` has not completed. A contrived example:: - - @interface A : NSObject { - NSString *_description; - } - - (id)init; - - (NSString*)description; - @end - - @implementation A - - (id)init { - self = [super init] - if (self) { - _description = [self description]; - } - return self; - } - - - (NSString *)description { - return @"A"; - } - @end - - @interface B : A - @property NSString *title; - @end - - @implementation B - - (id)init { - self = [super init] - if (self) { - self->title = @"Hello"; - } - return self; - } - - -(NSString *)description { - return self->title; - } - @end - -During the second phase of initialization, A's ``-init`` method -invokes the ``-description`` method, which ends up in B's -``-description``. Here, ``title`` will be ``nil`` even though the -apparent invariant (when looking at B's ``-init`` method) is that -``title`` is never nil, because B's ``-init`` method has not yet -finished execution. - -In a language with single-phase initialization (such as C++, Java, or -C# constructors), the object is considered to have the type of the -constructor currently executing for the purposes of dynamic -dispatch. For the Objective-C example above, this would mean that when -A's ``-init`` sends the ``description`` message, it would invoke A's -``-description``. This is somewhat safer than two-phase -initialization, because the programmer does not have to deal with the -possibility of executing one's methods before the initialization of -one's instance variables have completed. It is also less flexible. - -Designated Initializers -~~~~~~~~~~~~~~~~~~~~~~~ - -One of the benefits of Objective-C's ``init`` methods is that they are -instance methods, and therefore are inherited. One need not override -the ``init`` methods in a subclass unless that subclass has additional -instance variables that require initialization. However, when a -subclass does introduce additional instance variables that require -initialization, which ``init`` methods should it override? If the -answer is "all of them", then initializer inheritance isn't all that -useful. - -Objective-C convention has the notion of a `designated initializer`_ -which is, roughly, an initializer method that is responsible for -calling one of it's superclass's initializers, then initializing its -own instance variables. Initializers that are not designated -initializers are "secondary" initializers: they typically delegate to -another initializer (eventually terminating the chain at a designated -initializer) rather than performing initialization themselves. For -example, consider the following ``Task`` class (from the -aforementioned "Concepts in Objective-C Programming" document):: - - @interface Task - @property NSString *title; - @property NSDate *date; - - - (id)initWithTitle:(NSString *)aTitle date:(NSDate *)aDate; - - (id)initWithTitle:(NSString *)aTitle; - - (id)init; - @end - - @implementation Task - - (id)initWithTitle:(NSString *)aTitle date:(NSDate *)aDate { - title = aTitle; - date = aDate; - return self; - } - - - (id)initWithTitle:(NSString *)aTitle { - return [self initWithTitle:aTitle date:[NSDate date]]; - } - - - (id)init { - return [self initWithTitle:@"Task"]; - } - @end - -The first initializer is the designated initializer, which directly -initializes the instance variables from its parameters. The second two -initializers are secondary initializers, which delegate to other -initializers, eventually reaching the designated initializer. - -A subclass should override all of its superclass's designated -initializers, but it need not override the secondary initializers. We -can illustrate this with a subclass of ``Task`` that introduces a new -instance variable:: - - @interface PackagedTask : Task - @property dispatch_queue_t queue; - - - (id)initWithTitle:(NSString *)aTitle date:(NSDate *)aDate; - - (id)initWithTitle:(NSString *)aTitle date:(NSDate *)aDate queue:(dispatch_queue_t)aQueue; - @end - - @implementation PackagedTask - - (id)initWithTitle:(NSString *)aTitle date:(NSDate *)aDate { - return [self initWithTitle:aTitle - date:aDate - queue:dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)]; - } - - - (id)initWithTitle:(NSString * aTitle date:(NSDate *)aDate queue:(dispatch_queue_t)aQueue { - self = [super initWithTitle:aTitle date:aDate]; - if (self) { - queue = aQueue; - } - return self; - } - @end - -``PackagedTask`` overrides ``Task``'s designated initializer, -``-initWithTitle:date:``, which then becomes a secondary initializer -of ``PackagedTask``, whose only designated initializer is -``initWithTitle:date:queue:``. The latter method invokes its -superclass's designated initializer (``-[Task initWithTitle:date:]``), -then initializes its own instance variable. By following the rules of -designated initializers mentioned above, one ensures that the -inherited secondary initializers still work. Consider the execution -of ``[[PackagedTask alloc] init]``: - -* A ``PackagedTask`` object is allocated. -* ``-[Task init]`` executes, which delegates to ``-[Task - initWithTitle]``. -* ``-[Task initWithTitle]`` delegates to ``-initWithTitle:date:``, - which executes ``-[PackagedTask initWithTitle:date:]``. -* ``-[PackagedTask initWithTitle:date:]`` invokes ``-[Task - initWithTitle:date:]`` to initialize ``Task``'s instance variables, - then initializes its own instance variables. - -The Middle Phase of Initialization -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Objective-C's two-phase initialization actually has a third part, -which occurs between the zeroing of the instance variables and the -call to the initialization. This initialization invokes the default -constructors for instance variables of C++ class type when those -default constructors are not trivial. For example:: - - @interface A - struct X { - X() { printf("X()\n"); } - }; - - @interface A : NSObject { - X x; - } - @end - -When constructing an object with ``[[A alloc] init]``, the default -constructor for ``X`` will execute after the instance variables are -zeroed but before ``+alloc`` returns. - -Swift Constructors ------------------- - -Swift's constructors merge both allocation and initialization into a -single function. One constructs a new object with type construction -syntax as follows:: - - Task(title:"My task", date:NSDate()) - -The object will be allocated (via Swift's allocation routines) and the -corresponding constructor will be invoked to perform the -initialization. The constructor itself might look like this:: - - class Task { - var title : String - var date : NSDate - - constructor(title : String = "Task", date : NSDate = NSDate()) { - self.title = title - self.date = date - } - } - -Due to the use of default arguments, one can create a task an any of -the following ways:: - - Task() - Task(title:"My task") - Task(date:NSDate()) - Task(title:"My task", date:NSDate()) - -Constructor Delegation -~~~~~~~~~~~~~~~~~~~~~~ - -Although our use of default arguments has eliminated the need for the -various secondary constructors in the Objective-C version of the -``Task`` class, there are other reasons why one might want to have one -constructor in a class call another to perform initialization -(called constructor *delegation*). For example, default arguments -cannot make use of other argument values, and one may have a more -complicated default value. For example, the default title could depend -on the provided date. Swift should support constructor delegation for -this use case:: - - constructor(title : String, date : NSDate = NSDate()) { - self.title = title - self.date = date - } - - constructor(date : NSDate = NSDate()) { - /*self.*/constructor(title:"Task created on " + date.description(), - date:date) - } - -A constructor that delegates to another constructor must do so before -using or initializing any of its instance variables. This property can -be verified via definite initialization analysis. - -Superclass Constructors -~~~~~~~~~~~~~~~~~~~~~~~ - -When one class inherits another, each constructor within the subclass -must call one of its superclass's constructors before using or -initializing any of its instance variables, which is also verified via -definite initialization analysis. For example, the Swift -``PackagedTask`` class could be implemented as:: - - class PackagedTask : Task { - var queue : dispatch_queue_t - - constructor(title : String, date : NSDate = NSDate(), - queue : dispatch_queue_t = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)) { - super.constructor(title:title, date:date) - self.queue = queue - } - } - -Instance Variable Initialization -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Swift allows instance variables to be provided with an initializer, -which will be called as part of initialization of an object. For -example, we could provide initializers for the members of ``Task`` as -follows:: - - class Task { - var title : String = "Title" - var date : NSDate = NSDate() - } - -Here, one does not need to write any constructor: a default, -zero-parameter constructor will be synthesized by the compiler, which -runs the provided instance variable initializations. Similarly, if I -did write a constructor but did not initialize all of the instance -variables, each uninitialized instance variable would be initialized -by its provided initializer. While this is mainly programmer -convenience (one need only write the common initialization for an -instance variable once, rather than once per non-delegating -constructor), it may also have an impact on the overall initialization -story (see below). - - -One- or Two-Phase Initialization? -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Swift's current model it attempts to ensure that instance variables -are initialized before they are used via definite initialization -analysis. However, we haven't yet specified whether dynamic dispatch -during initialization sees the object as being of the -currently-executing constructor's type (as in C++/Java/C#) or of the -final subclass's type (as in Objective-C). - -I propose that we follow the C++/Java/C# precedent, which allows us to -ensure (within Swift code) that an instance variable is never accessed -before it has been initialized, eliminating the safety concerns -introduced by Objective-C's two-phase initialization. Practically -speaking, this means that the vtable/isa pointer should be set to the -constructor's type while the constructor executes. - -This model complicates constructor inheritance considerably. A -secondary initializer in Objective-C works by delegating to -(eventually) a designated initializer, which is overridden by the -subclass. Following the C++/Java/C# precedent breaks this pattern, -because the overriding designated initializer will never be invoked. - -Constructor Inheritance -~~~~~~~~~~~~~~~~~~~~~~~ - -Currently, constructors in Swift are not inherited. This is a -limitation that Swift's constructor model shares with C++98/03, Java, -and C#, and a regression from Objective-C. C++11 introduced the notion -of inherited constructors. It is an opt-in feature, introduced into -one's class with a using declaration such as:: - - using MySuperclass::MySuperclass; - -C++11 inherited constructors are implemented by essentially copying -the signatures of all of the superclass's constructors into the -subclass, ignoring those for which the subclass already has a -constructor with the same signature. This approach does not translate -well to Swift, because there is no way to gather all of the -constructors in either the superclass or the subclass due to the -presence of class extensions. - -One potential approach is to bring Objective-C's notion of designated -and secondary initializers into Swift. A "designated" constructor is -responsible for calling the superclass constructor and then -initializing its own instance variables. - -A "secondary" constructor can be written in the class definition or an -extension. A secondary constructor must delegate to another -constructor, which will find the constructor to delegate to based on -the type of the eventual subclass of the object. Thus, the subclass's -override of the designated constructor will initialize all of the -instance variables before continuing execution of the secondary -constructor. - -For the above to be safe, we need to ensure that subclasses -override all of the designated constructors of their -superclass. Therefore, we require that designated constructors be -written within the class definition [#]_. Secondary constructors can -be written in either the class definition or an -extension. - -In Objective-C, classes generally only have one or two designated -initializers, so having to override them doesn't seem too onerous. If -it begins to feel like boilerplate, we could perform the override -automatically when all of the instance variables of the subclass -themselves have initializers. - -Note that the requirement that all designated constructors be -initializable means that adding a new designated constructor to a -class is both an ABI- and API-breaking change. - -Syntactically, we could introduce some kind of attribute or keyword to -distinguish designated constructors from secondary -constructors. Straw men: ``[designated]`` for designated constructors, -which is implied for constructors in the class definition, and -``[inherited]`` for secondary constructors, which is implied for -constructors in class extensions. - -.. [#] We could be slightly more lenient here, allowing designated - constructors to be written in any class extension within the - same module as the class definition. - -Class Clusters and Assignment to Self -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -TBD. - - -Objective-C Interoperability ----------------------------- - -The proposed Swift model for constructors needs to interoperate well -with Objective-C, both for Objective-C classes imported into Swift and -for Swift classes imported into Objective-C. - -Constructor <-> Initializer Mapping -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Fundamental to the interoperability story is that Swift constructors -serve the same role as Objective-C initializers, and therefore should -be interoperable. An Objective-C object should be constructible with -Swift construction syntax, e.g.,:: - - NSDate() - -and one should be able to override an Objective-C initializer in a -Swift subclass by writing a constructor, e.g.,:: - - class Task : NSObject { - constructor () { - super.constructor() // invokes -[NSObject init] - - // perform initialization - } - } - -On the Objective-C side, one should be able to create a ``Task`` -object with ``[[Task alloc] init]``, subclass ``Task`` to override -``-init``, and so on. - -Each Swift constructor has both its two Swift entry points (one that -allocates the object, one that initializes it) and an Objective-C -entry point for the initializer. Given a Swift constructor, the -selector for the Objective-C entry point is formed by: - -* For the first selector piece, prepending the string ''init'' to the - capitalized name of the first parameter. -* For the remaining selector pieces, the names of the remaining - parameters. - -For example, given the Swift constructor:: - - constructor withTitle(aTitle : String) date(aDate : NSDate) { - // ... - } - -the Objective-C entry point will have the selector -``initWithTitle:date:``. Missing parameter names will be holes in the -selector (e.g., ''init::::''). If the constructor either has zero -parameters or if it has a single parameter with ``Void`` -type (i.e., the empty tuple ``()``), the selector will be a -zero-argument selector with no trailing colon, e.g.,:: - - constructor toMemory(_ : ()) { /* ... */ } - -maps to the selector ``initToMemory``. - -This mapping is reversible: given a selector in the “init” family, -i.e., where the first word is “init”, we split the selector into its -various pieces at the colons: - -* For the first piece, we remove the “init” and then lowercase the - next character *unless* the second character is also uppercase. This - becomes the name of the first parameter to the constructor. If this - string is non-empty and the selector is a zero-argument selector - (i.e., no trailing ``:``), the first (only) parameter has ``Void`` - type. -* For the remaining pieces, we use the selector piece as the name of - the corresponding parameter to the constructor. - -Note that this scheme intentionally ignores methods that don't have -the leading ``init`` keyword, including (e.g.) methods starting with -``_init`` or methods with completely different names that have been -tagged as being in the ``init`` family in Objective-C (via the -``objc_method_family`` attribute in Clang). We consider these cases to -be rare enough that we don't want to pessimize the conventional -``init`` methods to accommodate them. - -Designated Initializers -~~~~~~~~~~~~~~~~~~~~~~~ - -Designed initializers in Objective-C are currently identified by -documentation and convention. To strengthen these conventions and make -it possible to mechanically verify (in Swift) that all designated -initializers have been overridden, we should introduce a Clang method -attribute ``objc_designated_initializer`` that can be applied to the -designated initializers in Objective-C. Clang can then be extended to -perform similar checking to what we're describing for Swift: -designated initializers delegate or chain to the superclass -constructor, secondary constructors always delegate, and subclassing -rrequires one to override the designated initializers. The impact can -be softened somewhat using warnings or other heuristics, to be -(separately) determined. - -Nil and Re-assigned Self -~~~~~~~~~~~~~~~~~~~~~~~~ - -An Objective-C initializer can return a self pointer that is different -than the one it was called with. When this happens, it is either due -to an error (in which case it will return nil) or because the object -is being substituted for another object. - -In both cases, we are left with a partially-constructed object that -then needs to be destroyed, even though its instance variables may not -yet havee been initialized. This is also a problem for Objective-C, -which makes returning anything other than the original ''self'' -brittle. - -In Swift, we will have a separate error-handling mechanism to report -failures. A Swift constructor will not be allowed to return a value; -rather, it should raise an error if an error occurs, and that error -will be propagated however we eventually decide to implement error -propagation. - -Object substitution is the more complicated feature. I propose that we -do not initially support object substitution within Swift -constructors. However, for memory safety we do need to recognize when -calling the superclass constructor or delegating to another -constructor has replaced the object (which can happen in Objective-C -code). Aside from the need to destroy the original object, the -instance variables of the new object will already have been -initialized, so our "initialization" of those instance variables is -actually assignment. The code generation for constructors will need to -account for this. - -Alternatives ------------- - -This proposal is complicated, in part because it's trying to balance -the safety goals of Swift against the convience of Objective-C's -two-phase initialization. - -Separate Swift Constructors from Objective-C Initializers -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Rather than try to adapt Swift's constructors to work with -Objective-C's initializers, we could instead keep these features -distinct. Swift constructors would maintain their current behavior -(which ensures that instance variables are always initialized), and -neither override nor introduce Objective-C initializers as entry -points. - -In this world, one would still have to implement ``init`` methods in -Swift classes to override Objective-C initializers or make a Swift -object constructible in Objective-C via the ``alloc/init`` -pattern. Swift constructors would not be inherited, or would use some -mechanism like C++11's inherited constructors. As we do today, Swift's -Clang importer could introduce constructors into Objective-C classes -that simply forward to the underlying initializers, so that we get -Swift's object construction syntax. However, the need to write -``init`` methods when subclasses would feel like a kludge. - - -Embrace Two-Phase Initialization -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -We could switch Swift whole-heartedly over to two-phase -initialization, eliminating constructors in favor of ``init`` -methods. We would likely want to ensure that all instance variables -get initialized before the ``init`` methods ever run, for safety -reasons. This could be handled by (for example) requiring initializers -on all instance variables, and running those initializers after -allocation and before the ``init`` methods are called, the same way -that instance variables with non-trivial default constructors are -handled in Objective-C++. We would still likely need the notion of a -designated initializer in Objective-C to make this safe in Swift, -since we need to know which Objective-C initializers are guaranteed to -initialize those instance variables not written in Swift. - -This choice makes interoperability with Objective-C easier (since -we're adopting Objective-C's model), but it makes safety either harder -(e.g., we have to make all of our methods guard against uninitialized -instance variables) or more onerous (requiring initializers on the -declarations of all instance variables). - - - -.. _designated initializer: https://developer.apple.com/library/ios/documentation/general/conceptual/CocoaEncyclopedia/Initialization/Initialization.html diff --git a/docs/proposals/rejected/KeywordArguments.md b/docs/proposals/rejected/KeywordArguments.md new file mode 100644 index 0000000000000..dc4ebe46aa1bf --- /dev/null +++ b/docs/proposals/rejected/KeywordArguments.md @@ -0,0 +1,286 @@ +> **warning** +> +> This proposal was rejected. We explored a number of alternate + +> syntaxes for method names that both allowed describing arguments and +> were compatible with Objective-C selectors. + +Summary +======= + +We make the following incremental language changes: + +- We make it so that selector-style declaration syntax declares a + **selector name** for a method. `func foo(x:T) bar(y:U)` declares a + method named with a new selector reference syntax, `self.foo:bar:`. + This method cannot be referenced as `foo`; it can only be referenced + as `self.foo:bar:`, or with keyword application syntax, + `self.foo(x, bar: y)`. +- We change keywords in parens, as in `(a: x, b: y)` to be a feature + of apply syntax, instead of a feature of tuple literals. Name lookup + changes to resolve keywords according to the function declaration + rather than to the context tuple type. For tuple-style declarations + `func foo(a: Int, b: Int)`, keywords are optional and can + be reordered. For selector-style declarations + `func foo(a: Int) bar(b: Int)`, the keywords are required and must + appear in order. +- With keyword arguments no longer reliant on tuple types, we simplify + the tuple type system by removing keywords from tuple types. + +Keyword Apply Syntax +==================== + + expr-apply ::= expr-postfix '(' (kw-arg (',' kw-arg)*)? ')' + kw-arg ::= (identifier ':')? expr + + // Examples + foo() + foo(1, 2, 3) + foo(1, bar: 2, bas: 3) + foo!(1, bar: 2, bas: 3) + a.foo?(1, bar: 2, bas: 3) + +Keyword syntax becomes a feature of apply expressions. When a named +function or method is applied, a declaration is looked up using the name +and any keyword arguments that are provided. An arbitrary expression +that produces a function result may be applied, but cannot be used with +keyword arguments. + +Named Application Lookup +======================== + +A named application is matched to a declaration using both the base +name, which appears to the left of the open parens, and the keyword +arguments that are provided. The matching rules are different for +"tuple-style" and "keyword-style" declarations: + +Finding the base name +--------------------- + +The callee of an apply is an arbitrary expression. If the expression is +a standalone declaration reference or a member reference, then the apply +is a **named application**, and the name is used as the **base name** +during function name lookup, as described below. Certain expression +productions are looked through when looking for a base name: + +- The `Optional` postfix operators `!` and `?`. `foo.bar!()` and + `bas?()` are named applications, as is `foo?!?()`. +- Parens. `(foo.bar)()`\` is a named application. + +These productions are looked through to arbitrary depth, so +`((foo!?)?!)()` is also a named application. + +Unnamed applications cannot use keyword arguments. + +Tuple-Style Declarations +------------------------ + +A tuple-style declaration matches a named application if: + +- The base name matches the name of the declaration: + + func foo(x: Int, y: Int) {} + foo(1, 2) // matches + bar(1, 2) // doesn't match + +- Positional arguments, that is, arguments without keywords, must be + convertible to the type of the corresponding positional parameter of + the declaration: + + func foo(x: Int, y: Int) {} + foo(1, 2) // matches + foo("one", 2) // doesn't match + foo(1, "two") // doesn't match + foo("one", "two") // doesn't match + +- Keyword names, if present, must match the declared name of a + parameter in the declaration, and the corresponding argument must be + convertible to the type of the parameter in the declaration. The + same keyword name may not be used multiple times, and the same + argument cannot be provided positionally and by keyword. All keyword + arguments must follow all positional arguments: + + func foo(x: Int, y: String, z: UnicodeScalar) {} + foo(1, "two", '3') // matches + foo(1, "two", z: '3') // matches + foo(1, y: "two", '3') // invalid, positional arg after keyword arg + foo(1, z: '3', y: "two") // matches + foo(z: '3', x: 1, y: "two") // matches + foo(z: '3', q: 1, y: "two") // doesn't match; no keyword 'q' + foo(1, "two", '3', x: 4) // doesn't match; 'x' already given positionally + foo(1, "two", z: '3', z: '4') // doesn't match; multiple 'z' + + As an exception, a trailing closure argument always positionally + matches the last declared parameter, and can appear after keyword + arguments: + + func foo(x: Int, y: String, f: () -> () + foo(y: "two", x: 1) { } // matches + +- If the final declared keyword parameter takes a variadic argument, + the keyword in the argument list may be followed by multiple + non-keyworded arguments. All arguments up to either the next keyword + or the end of the argument list become that keyword's argument: + + func foo(x: Int, y: String, z: UnicodeScalar...) {} + foo(1, "two", '3', '4', '5') // matches, z = ['3', '4', '5'] + foo(1, "two", z: '3', '4', '5') // same + foo(1, z: '3', '4', '5', y: "two") // same + +- If the base name is the name of a non-function definition of + function type, the input types match the input type of the + referenced function value, and there are no keyword arguments: + + var foo: (Int, Int) -> () + foo(1, 2) // matches + foo(x: 1, y: 2) // doesn't match + +Selector-Style Declarations +--------------------------- + +A selector-style declaration matches a named application if: + +- The expression must provide keywords for all of its arguments but + the first. It must *not* provide a keyword for the first argument: + + func foo(x: Int) bar(y: String) bas(z: UnicodeScalar) {} + foo(1, "two", '3') // doesn't match; no keywords + foo(x: 1, bar: "two", bas: '3') // doesn't match; first keyword provided + foo(1, bar: "two", bas: '3') // matches + +- The base name of the apply expression must match the first declared + selector piece. The subsequent argument keyword names must match the + remaining selector pieces in order. The same keyword name may be + used multiple times, to refer to selector pieces with the same name. + The argument values must be convertible to the declared types of + each selector piece's parameter: + + func foo(x: Int) bar(y: String) bas(z: UnicodeScalar) {} + foo(1, bar: "two", bas: '3') // matches + foo(1, bas: '3', bar: "two") // doesn't match; wrong selector piece order + foo(1, bar: '2', bas: "three") // doesn't match; wrong types + + func foo(x: Int) foo(y: String) foo(z: UnicodeScalar) {} + foo(1, foo: "two", foo: '3') // matches + +- If the final selector piece declares a variadic parameter, then the + keyword in the call expression may be followed by + multiple arguments. All arguments up to the end of the argument list + become the keyword parameter's value. (Because of strict keyword + ordering, additional keywords may not follow.) For example: + + func foo(x: Int) bar(y: String...) {} + + foo(1, bar: "two", "three", "four") // matches, y = ["two", "three", "four"] + +- If the final selector piece declares a function parameter, then the + function can be called using trailing closure syntax omitting + the keyword. The keyword is still required when trailing closure + syntax is not used. For example: + + func foo(x: Int) withBlock(f: () -> ()) + + foo(1, withBlock: { }) // matches + foo(1, { }) // doesn't match + foo(1) { } // matches + + Trailing closure syntax can introduce ambiguities when + selector-style functions differ only in their final closure selector + piece: + + func foo(x: Int) onCompletion(f: () -> ()) + func foo(x: Int) onError(f: () -> ()) + + foo(1) { } // error: ambiguous + +Duplicate Definitions +===================== + +Tuple-Style Declarations +------------------------ + +Keyword names are part of a tuple-style declaration, but they are not +part of the declaration's name, they are not part of the declaration's +type, and they are not part of the declaration's ABI. Two tuple-style +declarations that differ only in keyword names are considered +duplicates: + + // Error: Duplicate definition of foo(Int, Int) -> () + func foo(a: Int, b: Int) {} + func foo(x: Int, y: Int) {} + +Selector-Style Declarations +--------------------------- + +The name of a selector-style declaration comprises all of its selector +pieces in declaration order. Selector-style declarations can be +overloaded by selector name, by selector order, and by type: + + // OK, no duplicates + func foo(x: Int) bar(y: Int) bas(z: Int) + func foo(x: Int) bar(y: Int) zim(z: Int) + func foo(x: Int) bas(y: Int) bar(z: Int) + func foo(x: Int) bar(y: Int) bas(z: Float) + +Tuple- and selector-style declarations are not considered duplicates, +even if they can match the same keywords with the same types: + + // OK, not duplicates + func foo(x: Int, bar: Int) + func foo(x: Int) bar(x: Int) + +Unapplied Name Lookup +===================== + +An unapplied declaration reference `identifier` or member reference +`obj.identifier` finds any tuple-style declaration whose name matches +the referenced name. It never finds selector-style declarations: + + func foo(a: Int, b: Int) {} + func foo(a: Int) bar(b: Int) {} + + var f = foo // Finds foo(Int, Int) -> (), not foo:bar: + +Selector Name Lookup +==================== + + expr-selector-member-ref ::= expr-postfix '.' identifier ':' (identifier ':')+ + +Unapplied selector-style declarations can be referenced as a member of +their enclosing context using selector member reference expressions. The +name must consist of at least two selector pieces, each followed by a +colon. (A single identifier followed by a colon, such as `foo.bar:`, is +parsed as a normal member reference `foo.bar` followed by a colon.) A +selector member reference expression finds any selector-style +declarations whose selector pieces match the named selector pieces in +order: + + class C { + func foo(a: Int) bar(b: Int) bas(c: Int) + func foo(a: Int) bas(b: Int) bar(c: Int) + + func foo(a: Int, bar: Int, bas: Int) + } + + var c: C + + c.foo:bar:bas: // Finds c.foo:bar:bas: (not c.foo or c.foo:bas:bar:) + c.foo:bas:bar: // Finds c.foo:bas:bar: + c.foo // Finds c.foo + +QoI Issues +========== + +Under this proposal, keyword resolution relies on being able to find a +named function declaration. This means that keywords cannot be used with +arbitrary expressions of function type. We however still need to parse +keywords in nameless applications for recovery. There are also +functional operators like `!` and `?` that we need to forward keyword +arguments through. Are there others? What about parens? `(foo)(bar: x)` +should probably work. + +This proposal also prevents a single-element name from being referenced +with selector syntax as `foo.bar:`. For QoI, we should recognize +attempts to reference a member in this way, such as +`if var f = foo.bar: {}` or `[foo.bar:: bas]`, and fixit away the +trailing colon. diff --git a/docs/proposals/rejected/KeywordArguments.rst b/docs/proposals/rejected/KeywordArguments.rst deleted file mode 100644 index ba238b78ae029..0000000000000 --- a/docs/proposals/rejected/KeywordArguments.rst +++ /dev/null @@ -1,278 +0,0 @@ -:orphan: - -.. warning:: This proposal was rejected. We explored a number of alternate - syntaxes for method names that both allowed describing arguments and were - compatible with Objective-C selectors. - -Summary -------- - -We make the following incremental language changes: - -- We make it so that selector-style declaration syntax declares a - **selector name** for a method. ``func foo(x:T) bar(y:U)`` declares a method - named with a new selector reference syntax, ``self.foo:bar:``. This method - cannot be referenced as ``foo``; it can only be referenced as - ``self.foo:bar:``, or with keyword application syntax, - ``self.foo(x, bar: y)``. - -- We change keywords in parens, as in ``(a: x, b: y)`` to be a feature of - apply syntax, instead of a feature of tuple literals. Name lookup changes to - resolve keywords according to the function declaration rather than to the - context tuple type. For tuple-style declarations ``func foo(a: Int, b: Int)``, - keywords are optional and can be reordered. For selector-style declarations - ``func foo(a: Int) bar(b: Int)``, the keywords are required and must appear - in order. - -- With keyword arguments no longer reliant on tuple types, we simplify the - tuple type system by removing keywords from tuple types. - -Keyword Apply Syntax --------------------- -:: - - expr-apply ::= expr-postfix '(' (kw-arg (',' kw-arg)*)? ')' - kw-arg ::= (identifier ':')? expr - - // Examples - foo() - foo(1, 2, 3) - foo(1, bar: 2, bas: 3) - foo!(1, bar: 2, bas: 3) - a.foo?(1, bar: 2, bas: 3) - -Keyword syntax becomes a feature of apply expressions. When a named -function or method is applied, a declaration is looked up using the name and any -keyword arguments that are provided. An arbitrary expression that produces -a function result may be applied, but cannot be used with keyword arguments. - -Named Application Lookup ------------------------- - -A named application is matched to a declaration using both the base name, which -appears to the left of the open parens, and the keyword arguments that are -provided. The matching rules are different for "tuple-style" and -"keyword-style" declarations: - -Finding the base name -````````````````````` - -The callee of an apply is an arbitrary expression. If the expression is a -standalone declaration reference or a member reference, then the apply is a -**named application**, and the name is used as the **base name** during function -name lookup, as described below. Certain expression productions are looked -through when looking for a base name: - -- The ``Optional`` postfix operators ``!`` and ``?``. - ``foo.bar!()`` and ``bas?()`` are named applications, as is ``foo?!?()``. -- Parens. ``(foo.bar)()``` is a named application. - -These productions are looked through to arbitrary depth, so ``((foo!?)?!)()`` -is also a named application. - -Unnamed applications cannot use keyword arguments. - -Tuple-Style Declarations -```````````````````````` - -A tuple-style declaration matches a named application if: - -- The base name matches the name of the declaration:: - - func foo(x: Int, y: Int) {} - foo(1, 2) // matches - bar(1, 2) // doesn't match - -- Positional arguments, that is, arguments without keywords, must be convertible - to the type of the corresponding positional parameter of the declaration:: - - func foo(x: Int, y: Int) {} - foo(1, 2) // matches - foo("one", 2) // doesn't match - foo(1, "two") // doesn't match - foo("one", "two") // doesn't match - -- Keyword names, if present, must match the declared name of a parameter in the - declaration, and the corresponding argument must be convertible to the type - of the parameter in the declaration. The same keyword name may not be used - multiple times, and the same argument cannot be provided positionally and - by keyword. All keyword arguments must follow all positional arguments:: - - func foo(x: Int, y: String, z: UnicodeScalar) {} - foo(1, "two", '3') // matches - foo(1, "two", z: '3') // matches - foo(1, y: "two", '3') // invalid, positional arg after keyword arg - foo(1, z: '3', y: "two") // matches - foo(z: '3', x: 1, y: "two") // matches - foo(z: '3', q: 1, y: "two") // doesn't match; no keyword 'q' - foo(1, "two", '3', x: 4) // doesn't match; 'x' already given positionally - foo(1, "two", z: '3', z: '4') // doesn't match; multiple 'z' - - As an exception, a trailing closure argument always positionally matches - the last declared parameter, and can appear after keyword arguments:: - - func foo(x: Int, y: String, f: () -> () - foo(y: "two", x: 1) { } // matches - -- If the final declared keyword parameter takes a variadic argument, the keyword - in the argument list may be followed by multiple - non-keyworded arguments. All arguments up to either the next keyword or - the end of the argument list become that keyword's argument:: - - func foo(x: Int, y: String, z: UnicodeScalar...) {} - foo(1, "two", '3', '4', '5') // matches, z = ['3', '4', '5'] - foo(1, "two", z: '3', '4', '5') // same - foo(1, z: '3', '4', '5', y: "two") // same - -- If the base name is the name of a non-function definition of function type, - the input types match the input type of the referenced function value, and - there are no keyword arguments:: - - var foo: (Int, Int) -> () - foo(1, 2) // matches - foo(x: 1, y: 2) // doesn't match - -Selector-Style Declarations -``````````````````````````` - -A selector-style declaration matches a named application if: - -- The expression must provide keywords for all of its arguments but the first. - It must *not* provide a keyword for the first argument:: - - func foo(x: Int) bar(y: String) bas(z: UnicodeScalar) {} - foo(1, "two", '3') // doesn't match; no keywords - foo(x: 1, bar: "two", bas: '3') // doesn't match; first keyword provided - foo(1, bar: "two", bas: '3') // matches - -- The base name of the apply expression must match the first declared selector - piece. The subsequent argument keyword names must match the remaining selector - pieces in order. The same keyword name may be used multiple times, to refer - to selector pieces with the same name. The argument values must be convertible - to the declared types of each selector piece's parameter:: - - func foo(x: Int) bar(y: String) bas(z: UnicodeScalar) {} - foo(1, bar: "two", bas: '3') // matches - foo(1, bas: '3', bar: "two") // doesn't match; wrong selector piece order - foo(1, bar: '2', bas: "three") // doesn't match; wrong types - - func foo(x: Int) foo(y: String) foo(z: UnicodeScalar) {} - foo(1, foo: "two", foo: '3') // matches - -- If the final selector piece declares a variadic parameter, then the keyword - in the call expression may be followed by multiple arguments. All arguments - up to the end of the argument list become the keyword parameter's value. - (Because of strict keyword ordering, additional keywords may not follow.) - For example:: - - func foo(x: Int) bar(y: String...) {} - - foo(1, bar: "two", "three", "four") // matches, y = ["two", "three", "four"] - -- If the final selector piece declares a function parameter, then the function - can be called using trailing closure syntax omitting the keyword. The keyword - is still required when trailing closure syntax is not used. For example:: - - func foo(x: Int) withBlock(f: () -> ()) - - foo(1, withBlock: { }) // matches - foo(1, { }) // doesn't match - foo(1) { } // matches - - Trailing closure syntax can introduce ambiguities when selector-style - functions differ only in their final closure selector piece:: - - func foo(x: Int) onCompletion(f: () -> ()) - func foo(x: Int) onError(f: () -> ()) - - foo(1) { } // error: ambiguous - -Duplicate Definitions ---------------------- - -Tuple-Style Declarations -```````````````````````` - -Keyword names are part of a tuple-style declaration, but they are not part -of the declaration's name, they are not part of the declaration's type, and -they are not part of the declaration's ABI. Two tuple-style declarations that -differ only in keyword names are considered duplicates:: - - // Error: Duplicate definition of foo(Int, Int) -> () - func foo(a: Int, b: Int) {} - func foo(x: Int, y: Int) {} - -Selector-Style Declarations -``````````````````````````` - -The name of a selector-style declaration comprises all of its selector pieces in -declaration order. Selector-style declarations can be overloaded by selector -name, by selector order, and by type:: - - // OK, no duplicates - func foo(x: Int) bar(y: Int) bas(z: Int) - func foo(x: Int) bar(y: Int) zim(z: Int) - func foo(x: Int) bas(y: Int) bar(z: Int) - func foo(x: Int) bar(y: Int) bas(z: Float) - -Tuple- and selector-style declarations are not considered duplicates, even if -they can match the same keywords with the same types:: - - // OK, not duplicates - func foo(x: Int, bar: Int) - func foo(x: Int) bar(x: Int) - -Unapplied Name Lookup ---------------------- - -An unapplied declaration reference ``identifier`` or member reference -``obj.identifier`` finds any tuple-style declaration whose name matches the -referenced name. It never finds selector-style declarations:: - - func foo(a: Int, b: Int) {} - func foo(a: Int) bar(b: Int) {} - - var f = foo // Finds foo(Int, Int) -> (), not foo:bar: - -Selector Name Lookup --------------------- -:: - - expr-selector-member-ref ::= expr-postfix '.' identifier ':' (identifier ':')+ - -Unapplied selector-style declarations can be referenced as a member of their -enclosing context using selector member reference expressions. The name must -consist of at least two selector pieces, each followed by a colon. (A single -identifier followed by a colon, such as ``foo.bar:``, is parsed as a normal -member reference ``foo.bar`` followed by a colon.) A selector member reference -expression finds any selector-style declarations whose selector pieces match the -named selector pieces in order:: - - class C { - func foo(a: Int) bar(b: Int) bas(c: Int) - func foo(a: Int) bas(b: Int) bar(c: Int) - - func foo(a: Int, bar: Int, bas: Int) - } - - var c: C - - c.foo:bar:bas: // Finds c.foo:bar:bas: (not c.foo or c.foo:bas:bar:) - c.foo:bas:bar: // Finds c.foo:bas:bar: - c.foo // Finds c.foo - -QoI Issues ----------- - -Under this proposal, keyword resolution relies on being able to find a named -function declaration. This means that keywords cannot be used with arbitrary -expressions of function type. -We however still need to parse keywords in nameless applications for recovery. -There are also functional operators like ``!`` and ``?`` that we need to -forward keyword arguments through. Are there others? What about parens? -``(foo)(bar: x)`` should probably work. - -This proposal also prevents a single-element name from being referenced with -selector syntax as ``foo.bar:``. For QoI, we should recognize attempts to -reference a member in this way, such as ``if var f = foo.bar: {}`` or -``[foo.bar:: bas]``, and fixit away the trailing colon. diff --git a/docs/proposals/valref.md b/docs/proposals/valref.md new file mode 100644 index 0000000000000..9bbc52efd28ce --- /dev/null +++ b/docs/proposals/valref.md @@ -0,0 +1,554 @@ +Values and References +===================== + +Author + +: Dave Abrahams + +Author + +: Joe Groff + +Date + +: 2013-03-15 + +**Abstract:** We propose a system that offers first-class support for +both value and reference semantics. By allowing—but not +requiring—(instance) variables, function parameters, and generic +constraints to be declared as `val` or `ref`, we offer users the ability +to nail down semantics to the desired degree without compromising ease +of use. + +> **note** +> +> We are aware of some issues with naming of these new keywords; to +> avoid chaos we discuss alternative spelling schemes in a Bikeshed\_ +> section at the end of this document. + +Introduction +------------ + +Until recently, Swift's support for value semantics outside trivial +types like scalars and immutable strings has been weak. While the recent +`Clonable` proposal makes new things possible in the “safe” zone, it +leaves the language syntactically and semantically lumpy, keeping +interactions between value and reference types firmly outside the “easy” +zone and failing to address the issue of generic programming. + +This proposal builds on the `Clonable` proposal to create a more +uniform, flexible, and interoperable type system while solving the +generic programming problem and expanding the “easy” zone. + +General Description +------------------- + +The general rule we propose is that most places where you can write +`var` in today's swift, and also on function parameters, you can write +`val` or `ref` to request value or reference semantics, respectively. +Writing `var` requests the default semantics for a given type. +Non-`class` types (`struct`s, tuples, arrays, `union`s) default to `val` +semantics, while `class`es default to `ref` semantics. The types +`val SomeClass` and `ref SomeStruct` also become part of the type system +and can be used as generic parameters or as parts of tuple, array, and +function types. + +Because the current specification already describes the default +behaviors, we will restrict ourselves to discussing the new +combinations, such as `struct` variables declared with `ref` and `class` +variables declared with `val`, and interactions between the two. + +Terminology +----------- + +When we use the term "copy" for non-`class` types, we are talking about +what traditionally happens on assignment and pass-by-value. When applied +to `class` types, "copy" means to call the `clone()` method, which is +generated by the compiler when the user has explicitly declared +conformance to the `Clonable` protocol. + +When we refer to variables being “declared `val`” or “declared `ref`”, +we mean to include the case of equivalent declarations using `var` that +request the default semantics for the type. + +Unless otherwise specified, we discuss implementation details such as +"allocated on the heap" as a way of describing operational semantics, +with the understanding that semantics-preserving optimizations are +always allowed. + +When we refer to the "value" of a class, we mean the combination of +values of its `val` instance variables and the identities of its `ref` +instance variables. + +Variables +--------- + +Variables can be explicitly declared `val` or `ref`: + + var x: Int // x is stored by value + val y: Int // just like "var y: Int" + ref z: Int // z is allocated on the heap. + + var q: SomeClass // a reference to SomeClass + ref r: SomeClass // just like "var r: SomeClass" + val s: SomeClonableClass // a unique value of SomeClonableClass type + +Assignments and initializations involving at least one `val` result in a +copy. Creating a `ref` from a `val` copies into heap memory: + + ref z2 = x // z2 is a copy of x's value on the heap + y = z // z2's value is copied into y + + ref z2 = z // z and z2 refer to the same Int value + ref z3 = z.clone() // z3 refers to a copy of z's value + + val t = r // Illegal unless SomeClass is Clonable + ref u = s // s's value is copied into u + val v = s // s's value is copied into v + +Standalone Types +---------------- + +`val`- or `ref`-ness is part of the type. When the type appears without +a variable name, it can be written this way: + + ref Int // an Int on the heap + val SomeClonableClass // a value of SomeClonableClass type + +Therefore, although it is not recommended style, we can also write: + + var y: val Int // just like "var y: Int" + var z: ref Int // z is allocated on the heap. + var s: val SomeClonableClass // a unique value of type SomeClonableClass + +Instance Variables +------------------ + +Instance variables can be explicitly declared `val` or `ref`: + + struct Foo { + var x: Int // x is stored by-value + val y: Int // just like "var y: Int" + ref z: Int // allocate z on the heap + + var q: SomeClass // q is a reference to SomeClass + ref r: SomeClass // just like "var r: SomeClass" + val s: SomeClonableClass // clone() s when Foo is copied + } + + class Bar : Clonable { + var x: Int // x is stored by-value + val y: Int // just like "var y: Int" + ref z: Int // allocate z on the heap + + var q: SomeClass // q is stored by-reference + ref r: SomeClass // just like "var r: SomeClass" + val s: SomeClonableClass // clone() s when Bar is clone()d + } + +When a value is copied, all of its instance variables declared `val` +(implicitly or explicitly) are copied. Instance variables declared `ref` +merely have their reference counts incremented (i.e. the reference is +copied). Therefore, when the defaults are in play, the semantic rules +already defined for Swift are preserved. + +The new rules are as follows: + +- A non-`class` instance variable declared `ref` is allocated on the + heap and can outlive its enclosing `struct`. +- A `class` instance variable declared `val` will be copied when its + enclosing `struct` or `class` is copied. We discuss + [below](%60standalone%20types%60_) what to do when the `class` is + not `Clonable`. + +Arrays +------ + +TODO: reconsider sugared array syntax. Maybe val<Int>\[42\] would +be better + +Array elements can be explicitly declared `val` or `ref`: + + var x : Int[42] // an array of 42 integers + var y : Int[val 42] // an array of 42 integers + var z : Int[ref 42] // an array of 42 integers-on-the-heap + var z : Int[ref 2][42] // an array of 2 references to arrays + ref a : Int[42] // a reference to an array of 42 integers + +When a reference to an array appears without a variable name, it can be +written using the [usual syntax](non-copyable_): + + var f : ()->ref Int[42] // a closure returning a reference to an array + var b : ref Int[42] // equivalent to to "ref b : Int[42]" + +Presumably there is also some fully-desugared syntax using angle +brackets, that most users will never touch, e.g.: + + var x : Array // an array of 42 integers + var y : Array // an array of 42 integers + var z : Array // an array of 42 integers-on-the-heap + var z : Array, 2> // an array of 2 references to arrays + ref a : Array // a reference to an array of 42 integers + var f : ()->ref Array // a closure returning a reference to an array + var b : ref Array // equivalent to to "ref b : Int[42]" + +Rules for copying array elements follow those of instance variables. + +`union`s +-------- + +Union types, like structs, have default value semantics. Constructors +for the union can declare the `val`- or `ref`-ness of their associated +values, using the same syntax as function parameters, described below: + + union Foo { + case Bar(ref bar:Int) + case Bas(val bas:SomeClass) + } + +Unions allow the definition of recursive types. A constructor for a +union may recursively reference the union as a member; the necessary +indirection and heap allocation of the recursive data structure is +implicit and has value semantics: + + // A list with value semantics--copying the list recursively copies the + // entire list + union List { + case Nil() + case Cons(car:T, cdr:List) + } + + // A list node with reference semantics—copying the node creates a node + // that shares structure with the tail of the list + union Node { + case Nil() + case Cons(car:T, ref cdr:Node) + } + +A special `union` type is the nullable type `T?`, which is sugar syntax +for a generic union type `Nullable`. Since both nullable refs and +refs-that-are-nullable are useful, we could provide sugar syntax for +both to avoid requiring parens: + + ref? Int // Nullable reference to Int: Nullable + ref Int? // Reference to nullable Int: ref Nullable + val? SomeClass // Nullable SomeClass value: Nullable + val Int? // nullable Int: val Nullable -- the default for Nullable + +Function Parameters +------------------- + +Function parameters can be explicitly declared `val`, or `ref`: + + func baz( + x: Int // x is passed by-value + , val y: Int // just like "y: Int" + , ref z: Int // allocate z on the heap + + , q: SomeClass // passing a reference + , ref r: SomeClass // just like "var r: SomeClass" + , val s: SomeClonableClass) // Passing a copy of the argument + +> **note** +> +> We suggest allowing explicit `var` function parameters for +> +> : uniformity. +> +Semantics of passing arguments to functions follow those of assignments +and initializations: when a `val` is involved, the argument value is +copied. + +> **note** +> +> We believe that `[inout]` is an independent concept and still very +> much needed, even with an explicit `ref` keyword. See also the +> Bikeshed\_ discussion at the end of this document. + +Generics +-------- + +TODO: Why do we need these constraints? TODO: Consider generic +classes/structs + +As with an array's element type, a generic type parameter can also be +bound to a `ref` or a `val` type. + +> var rv = new Array<ref Int> // Create a vector of +> Ints-on-the-heap var vv = new Array<val SomeClass> // Create a +> vector that owns its SomeClasses + +The rules for declarations in terms of `ref` or `val` types are that an +explicit `val` or `ref` overrides any `val`- or `ref`-ness of the type +parameter, as follows: + + ref x : T // always declares a ref + val x : T // always declares a val + var x : T // declares a val iff T is a val + +`ref` and `val` can be specified as protocol constraints for type +parameters: + + // Fill an array with independent copies of x + func fill(array:T[], x:T) { + for i in 0...array.length { + array[i] = x + } + } + +Protocols similarly can inherit from `val` or `ref` constraints, to +require conforming types to have the specified semantics: + + protocol Disposable : ref { + func dispose() + } + +The ability to explicitly declare `val` and `ref` allow us to smooth out +behavioral differences between value and reference types where they +could affect the correctness of algorithms. The continued existence of +`var` allows value-agnostic generic algorithms, such as `swap`, to go on +working as before. + +Non-Copyability +--------------- + +A non-`Clonable` `class` is not copyable. That leaves us with several +options: + +1. Make it illegal to declare a non-copyable `val` +2. Make non-copyable `val`s legal, but not copyable, thus infecting + their enclosing object with non-copyability. +3. Like \#2, but also formalize move semantics. All `val`s, including + non-copyable ones, would be explicitly movable. Generic `var` + parameters would probably be treated as movable but non-copyable. + +We favor taking all three steps, but it's useful to know that there are +valid stopping points along the way. + +Default Initialization of ref +----------------------------- + +TODO + +Array +----- + +TODO: Int\[...\], etc. + +Equality and Identity +--------------------- + +TODO + +Why Expand the Type System? +--------------------------- + +TODO + +Why do We Need `[inout]` if we have `ref`? +------------------------------------------ + +TODO + +Why Does the Outer Qualifier Win? +--------------------------------- + +TODO + +Objective-C Interoperability +---------------------------- + +### Clonable Objective-C classes + +In Cocoa, a notion similar to clonability is captured in the `NSCopying` +and `NSMutableCopying` protocols, and a notion similar to `val` instance +variables is captured by the behavior of `(copy)` properties. However, +there are some behavioral and semantic differences that need to be taken +into account. `NSCopying` and `NSMutableCopying` are entangled with +Foundation's idiosyncratic management of container mutability: +`-[NSMutableThing copy]` produces a freshly copied immutable `NSThing`, +whereas `-[NSThing copy]` returns the same object back if the receiver +is already immutable. `-[NSMutableThing mutableCopy]` and +`-[NSThing mutableCopy]` both return a freshly copied `NSMutableThing`. +In order to avoid requiring special case Foundation-specific knowledge +of whether class types are notionally immutable or mutable, we propose +this first-draft approach to mapping the Cocoa concepts to `Clonable`: + +- If an Objective-C class conforms to `NSMutableCopying`, use the + `-mutableCopyWithZone:` method to fulfill the Swift `Clonable` + concept, casting the result of `-mutableCopyWithZone:` back to the + original type. +- If an Objective-C class conforms to `NSCopying` but not + `NSMutableCopying`, use `-copyWithZone:`, also casting the result + back to the original type. + +This is suboptimal for immutable types, but should work for any Cocoa +class that fulfills the `NSMutableCopying` or `NSCopying` contracts +without requiring knowledge of the intended semantics of the class +beyond what the compiler can see. + +Objective-C `(copy)` properties should behave closely enough to Swift +`val` properties to be able to vend Objective-C `(copy)` properties to +Swift as `val` properties, and vice versa. + +### Objective-C protocols + +In Objective-C, only classes can conform to protocols, and the `This` +type is thus presumed to have references semantics. Swift protocols +imported from Objective-C or declared as `[objc]` could be conformed to +by `val` types, but doing so would need to incur an implicit copy to the +heap to create a `ref` value to conform to the protocol. + +How This Design Improves Swift +------------------------------ + +1. You can choose semantics at the point of use. The designer of a type + doesn't know whether you will want to use it via a reference; she + can only guess. You might *want* to share a reference to a struct, + tuple, etc. You might *want* some class type to be a component of + the value of some other type. We allow that, without requiring + awkward explicit wrapping, and without discarding the obvious + defaults for types that have them. +2. We provide a continuum of strictness in which to program. If you're + writing a script, you can go with `var` everywhere: don't worry; + be happy. If you're writing a large-scale program and want to be + very sure of what you're getting, you can forbid `var` except in + carefully-vetted generic functions. The choice is yours. +3. We allow generic programmers to avoid subtle semantic errors by + explicitly specifying value or reference semantics where it matters. +4. We move the cases where values and references interact much closer + to, and arguably into, the “easy” zone. + +How This Design Beats Rust/C++/C\#/etc. +--------------------------------------- + +- Simple programs stay simple. Rust has a great low-level memory + safety story, but it comes at the expense of ease-of-use. You can't + learn to use that system effectively without confronting three + [kinds](http://static.rust-lang.org/doc/tutorial.html#boxes-and-pointers) + of pointer, [named + lifetimes](http://static.rust-lang.org/doc/tutorial-borrowed-ptr.html#named-lifetimes), + [borrowing managed boxes and + rooting](http://static.rust-lang.org/doc/tutorial-borrowed-ptr.html#borrowing-managed-boxes-and-rooting), etc. + By contrast, there's a path to learning swift that postpones the + `val`/`ref` distinction, and that's pretty much *all* one must learn + to have a complete understanding of the object model in the “easy” + and “safe” zones. +- Simple programs stay safe. C++ offers great control over everything, + but the sharp edges are always exposed. This design allows + programmers to accomplish most of what people want to with C++, but + to do it safely and expressively. As with the rest of Swift, the + sharp edges are still available as an opt-in feature, and without + harming the rest of the language. +- Unlike C++, types meant to be reference types, supporting + inheritance, aren't copyable by default. This prevents inadvertent + slicing and wrong semantics. +- By retaining the `class` vs. `struct` distinction, we give type + authors the ability to provide a default semantics for their types + and avoid confronting their users with a constant `T*` vs. `T` + choice like C/C++. +- C\# also provides a `class` vs. `struct` distinction with a generics + system, but it provides no facilities for nontrivial value semantics + on struct types, and the only means for writing generic algorithms + that rely on value or reference semantics is to apply a blunt + `struct` or `class` constraint to type parameters and limit the type + domain of the generic. By generalizing both value and reference + semantics to all types, we allow both for structs with interesting + value semantics and for generics that can reliably specify and use + value or reference semantics without limiting the types they can be + used with. + +`structs` Really Should Have Value Semantics +-------------------------------------------- + +It is *possible* to build a struct with reference semantics. For +example, + +..parsed-literal: + + struct XPair + { + constructor() { + // These Xs are notionally **part of my value** + first = new X + second = new X + } + **ref** first : X + **ref** second : X + } + +However, the results can be surprising: + +If `XPair` had been declared a class, : + + val a : XPair // I want an independent value, please! + +would only compile if `XPair` was also `Clonable`, thereby protecting +the user's intention to create an independent value + +Getting the `ref` out of a `class` instance declared `val` +---------------------------------------------------------- + +A `class` instance is always accessed through a reference, but when an +instance is declared `val`, that reference is effectively hidden behind +the `val` wrapper. However, because `this` is passed to `class` methods +as a reference, we can unwrap the underlying `ref` as follows: + + val x : SomeClass + + extension SomeClass { + func get_ref() { return this } + } + + ref y : x.get_ref() + y.mutate() // mutates x + +Teachability +------------ + +By expanding the type system we have added complexity to the language. +To what degree will these changes make Swift harder to learn? + +We believe the costs can be mitigated by teaching plain `var` +programming first. The need to confront `val` and `ref` can be postponed +until the point where students must see them in the interfaces of +library functions. All the same standard library interfaces that could +be expressed before the introduction of `val` and `ref` can still be +expressed without them, so this discovery can happen arbitrarily late in +the game. However, it's important to realize that having `val` and `ref` +available will probably change the optimal way to express the standard +library APIs, and choosing where to use the new capabilities may be an +interesting balancing act. + +(Im)Mutability +-------------- + +We have looked, but so far, we don't think this proposal closes (or, for +that matter, opens) the door to anything fundamentally new with respect +to declared (im)mutability. The issues that arise with explicit `val` +and `ref` also arise without them. + +Bikeshed +-------- + +There are a number of naming issues we might want to discuss. For +example: + +- `var` is only one character different from `val`. Is that too + confusable? Syntax highlighting can help, but it might not + be enough. + - What about `let` as a replacement for `var`? There's always the + dreaded `auto`. + - Should we drop `let`/`var`/`auto` for ivars, because it “just + feels wrong” there? +- `ref` is spelled like `[inout]`, but they mean very different things + - We don't think they can be collapsed into one keyword: `ref` + requires shared ownership and is escapable and aliasable, unlike + `[inout]`. + - Should we spell `[inout]` differently? I think at a high level + it means something like “`[rebind]` the name to a new value.” +- Do we want to consider replacing `struct` and/or `class` with new + names such as `valtype` and `reftype`? We don't love those + particular suggestions. One argument in favor of a change: `struct` + comes with a strong connotation of weakness or second-class-ness for + some people. diff --git a/docs/proposals/valref.rst b/docs/proposals/valref.rst deleted file mode 100644 index 0d69d0f82e6ba..0000000000000 --- a/docs/proposals/valref.rst +++ /dev/null @@ -1,578 +0,0 @@ -:orphan: - -.. @raise litre.TestsAreMissing -.. _valref: - -======================= - Values and References -======================= - -:Author: Dave Abrahams -:Author: Joe Groff -:Date: 2013-03-15 - -**Abstract:** We propose a system that offers first-class support for -both value and reference semantics. By allowing—but not -requiring—(instance) variables, function parameters, and generic -constraints to be declared as ``val`` or ``ref``, we offer users the -ability to nail down semantics to the desired degree without -compromising ease of use. - -.. Note:: - - We are aware of some issues with naming of these new keywords; to - avoid chaos we discuss alternative spelling schemes in a Bikeshed_ - section at the end of this document. - -Introduction -============ - -Until recently, Swift's support for value semantics outside trivial -types like scalars and immutable strings has been weak. While the -recent ``Clonable`` proposal makes new things possible in the “safe” -zone, it leaves the language syntactically and semantically lumpy, -keeping interactions between value and reference types firmly outside -the “easy” zone and failing to address the issue of generic -programming. - -This proposal builds on the ``Clonable`` proposal to create a more -uniform, flexible, and interoperable type system while solving the -generic programming problem and expanding the “easy” zone. - -General Description -=================== - -The general rule we propose is that most places where you can write -``var`` in today's swift, and also on function parameters, you can -write ``val`` or ``ref`` to request value or reference semantics, -respectively. Writing ``var`` requests the default semantics for a -given type. Non-``class`` types (``struct``\ s, tuples, arrays, -``union``\ s) default to ``val`` semantics, while ``class``\ es -default to ``ref`` semantics. The types ``val SomeClass`` and -``ref SomeStruct`` also become part of the type system and can -be used as generic parameters or as parts of tuple, array, and -function types. - -Because the current specification already describes the default -behaviors, we will restrict ourselves to discussing the new -combinations, such as ``struct`` variables declared with ``ref`` and -``class`` variables declared with ``val``, and interactions between -the two. - -Terminology -=========== - -When we use the term "copy" for non-``class`` types, we are talking -about what traditionally happens on assignment and pass-by-value. -When applied to ``class`` types, "copy" means to call the ``clone()`` -method, which is generated by the compiler when the user has -explicitly declared conformance to the ``Clonable`` protocol. - -When we refer to variables being “declared ``val``” or “declared -``ref``”, we mean to include the case of equivalent declarations using -``var`` that request the default semantics for the type. - -Unless otherwise specified, we discuss implementation details such as -"allocated on the heap" as a way of describing operational semantics, -with the understanding that semantics-preserving optimizations are -always allowed. - -When we refer to the "value" of a class, we mean the combination of -values of its ``val`` instance variables and the identities of its -``ref`` instance variables. - -Variables -========= - -Variables can be explicitly declared ``val`` or ``ref``:: - - var x: Int // x is stored by value - val y: Int // just like "var y: Int" - ref z: Int // z is allocated on the heap. - - var q: SomeClass // a reference to SomeClass - ref r: SomeClass // just like "var r: SomeClass" - val s: SomeClonableClass // a unique value of SomeClonableClass type - -Assignments and initializations involving at least one ``val`` result -in a copy. Creating a ``ref`` from a ``val`` copies into heap memory:: - - ref z2 = x // z2 is a copy of x's value on the heap - y = z // z2's value is copied into y - - ref z2 = z // z and z2 refer to the same Int value - ref z3 = z.clone() // z3 refers to a copy of z's value - - val t = r // Illegal unless SomeClass is Clonable - ref u = s // s's value is copied into u - val v = s // s's value is copied into v - -Standalone Types -================ - -``val``\ - or ``ref``\ -ness is part of the type. When the type -appears without a variable name, it can be written this way:: - - ref Int // an Int on the heap - val SomeClonableClass // a value of SomeClonableClass type - -Therefore, although it is not recommended style, we can also write:: - - var y: val Int // just like "var y: Int" - var z: ref Int // z is allocated on the heap. - var s: val SomeClonableClass // a unique value of type SomeClonableClass - -Instance Variables -================== - -Instance variables can be explicitly declared ``val`` or ``ref``:: - - struct Foo { - var x: Int // x is stored by-value - val y: Int // just like "var y: Int" - ref z: Int // allocate z on the heap - - var q: SomeClass // q is a reference to SomeClass - ref r: SomeClass // just like "var r: SomeClass" - val s: SomeClonableClass // clone() s when Foo is copied - } - - class Bar : Clonable { - var x: Int // x is stored by-value - val y: Int // just like "var y: Int" - ref z: Int // allocate z on the heap - - var q: SomeClass // q is stored by-reference - ref r: SomeClass // just like "var r: SomeClass" - val s: SomeClonableClass // clone() s when Bar is clone()d - } - -When a value is copied, all of its instance variables declared ``val`` -(implicitly or explicitly) are copied. Instance variables declared -``ref`` merely have their reference counts incremented (i.e. the -reference is copied). Therefore, when the defaults are in play, the -semantic rules already defined for Swift are preserved. - -The new rules are as follows: - -* A non-``class`` instance variable declared ``ref`` is allocated on - the heap and can outlive its enclosing ``struct``. - -* A ``class`` instance variable declared ``val`` will be copied when - its enclosing ``struct`` or ``class`` is copied. We discuss below__ - what to do when the ``class`` is not ``Clonable``. - -Arrays -====== - -TODO: reconsider sugared array syntax. Maybe val[42] would be better - -Array elements can be explicitly declared ``val`` or ``ref``:: - - var x : Int[42] // an array of 42 integers - var y : Int[val 42] // an array of 42 integers - var z : Int[ref 42] // an array of 42 integers-on-the-heap - var z : Int[ref 2][42] // an array of 2 references to arrays - ref a : Int[42] // a reference to an array of 42 integers - -When a reference to an array appears without a variable name, it can -be written using the `usual syntax`__:: - - var f : ()->ref Int[42] // a closure returning a reference to an array - var b : ref Int[42] // equivalent to to "ref b : Int[42]" - -__ `standalone types`_ - -Presumably there is also some fully-desugared syntax using angle -brackets, that most users will never touch, e.g.:: - - var x : Array // an array of 42 integers - var y : Array // an array of 42 integers - var z : Array // an array of 42 integers-on-the-heap - var z : Array, 2> // an array of 2 references to arrays - ref a : Array // a reference to an array of 42 integers - var f : ()->ref Array // a closure returning a reference to an array - var b : ref Array // equivalent to to "ref b : Int[42]" - -Rules for copying array elements follow those of instance variables. - -``union``\ s -============ - -Union types, like structs, have default value semantics. Constructors for the -union can declare the ``val``- or ``ref``-ness of their associated values, using -the same syntax as function parameters, described below:: - - union Foo { - case Bar(ref bar:Int) - case Bas(val bas:SomeClass) - } - -Unions allow the definition of recursive types. A constructor for a union -may recursively reference the union as a member; the necessary -indirection and heap allocation of the recursive data structure is implicit -and has value semantics:: - - // A list with value semantics--copying the list recursively copies the - // entire list - union List { - case Nil() - case Cons(car:T, cdr:List) - } - - // A list node with reference semantics—copying the node creates a node - // that shares structure with the tail of the list - union Node { - case Nil() - case Cons(car:T, ref cdr:Node) - } - -A special ``union`` type is the nullable type ``T?``, which is -sugar syntax for a generic union type ``Nullable``. Since both nullable -refs and refs-that-are-nullable are useful, we could provide sugar syntax for -both to avoid requiring parens:: - - ref? Int // Nullable reference to Int: Nullable - ref Int? // Reference to nullable Int: ref Nullable - val? SomeClass // Nullable SomeClass value: Nullable - val Int? // nullable Int: val Nullable -- the default for Nullable - -__ non-copyable_ - -Function Parameters -=================== - -Function parameters can be explicitly declared ``val``, or ``ref``:: - - func baz( - x: Int // x is passed by-value - , val y: Int // just like "y: Int" - , ref z: Int // allocate z on the heap - - , q: SomeClass // passing a reference - , ref r: SomeClass // just like "var r: SomeClass" - , val s: SomeClonableClass) // Passing a copy of the argument - -.. Note:: We suggest allowing explicit ``var`` function parameters for - uniformity. - -Semantics of passing arguments to functions follow those of -assignments and initializations: when a ``val`` is involved, the -argument value is copied. - -.. Note:: - - We believe that ``[inout]`` is an independent concept and still very - much needed, even with an explicit ``ref`` keyword. See also the - Bikeshed_ discussion at the end of this document. - -Generics -======== - -TODO: Why do we need these constraints? -TODO: Consider generic classes/structs - -As with an array's element type, a generic type parameter can also be bound to -a ``ref`` or a ``val`` type. - - var rv = new Array // Create a vector of Ints-on-the-heap - var vv = new Array // Create a vector that owns its SomeClasses - -The rules for declarations in terms of ``ref`` or ``val`` types are that -an explicit ``val`` or ``ref`` overrides any ``val``- or ``ref``-ness of the -type parameter, as follows:: - - ref x : T // always declares a ref - val x : T // always declares a val - var x : T // declares a val iff T is a val - -``ref`` and ``val`` can be specified as protocol constraints for type -parameters:: - - // Fill an array with independent copies of x - func fill(array:T[], x:T) { - for i in 0...array.length { - array[i] = x - } - } - -Protocols similarly can inherit from ``val`` or ``ref`` constraints, to require -conforming types to have the specified semantics:: - - protocol Disposable : ref { - func dispose() - } - -The ability to explicitly declare ``val`` and ``ref`` allow us to -smooth out behavioral differences between value and reference types -where they could affect the correctness of algorithms. The continued -existence of ``var`` allows value-agnostic generic algorithms, such as -``swap``, to go on working as before. - -.. _non-copyable: - -Non-Copyability -=============== - -A non-``Clonable`` ``class`` is not copyable. That leaves us with -several options: - -1. Make it illegal to declare a non-copyable ``val`` -2. Make non-copyable ``val``\ s legal, but not copyable, thus - infecting their enclosing object with non-copyability. -3. Like #2, but also formalize move semantics. All ``val``\ s, - including non-copyable ones, would be explicitly movable. Generic - ``var`` parameters would probably be treated as movable but - non-copyable. - -We favor taking all three steps, but it's useful to know that there -are valid stopping points along the way. - -Default Initialization of ref -============================= - -TODO - -Array -===== - -TODO: Int[...], etc. - -Equality and Identity -===================== - -TODO - -Why Expand the Type System? -=========================== - -TODO - -Why do We Need ``[inout]`` if we have ``ref``? -============================================== - -TODO - -Why Does the Outer Qualifier Win? -================================= - -TODO - - -Objective-C Interoperability -============================ - -Clonable Objective-C classes ------------------------------ - -In Cocoa, a notion similar to clonability is captured in the ``NSCopying`` and -``NSMutableCopying`` protocols, and a notion similar to ``val`` instance -variables is captured by the behavior of ``(copy)`` properties. However, there -are some behavioral and semantic differences that need to be taken into account. -``NSCopying`` and ``NSMutableCopying`` are entangled with Foundation's -idiosyncratic management of container mutability: ``-[NSMutableThing copy]`` -produces a freshly copied immutable ``NSThing``, whereas ``-[NSThing copy]`` -returns the same object back if the receiver is already immutable. -``-[NSMutableThing mutableCopy]`` and ``-[NSThing mutableCopy]`` both return -a freshly copied ``NSMutableThing``. In order to avoid requiring special case -Foundation-specific knowledge of whether class types are notionally immutable -or mutable, we propose this first-draft approach to mapping the Cocoa concepts -to ``Clonable``: - -* If an Objective-C class conforms to ``NSMutableCopying``, use the - ``-mutableCopyWithZone:`` method to fulfill the Swift ``Clonable`` concept, - casting the result of ``-mutableCopyWithZone:`` back to the original type. -* If an Objective-C class conforms to ``NSCopying`` but not ``NSMutableCopying``, - use ``-copyWithZone:``, also casting the result back to the original type. - -This is suboptimal for immutable types, but should work for any Cocoa class -that fulfills the ``NSMutableCopying`` or ``NSCopying`` contracts without -requiring knowledge of the intended semantics of the class beyond what the -compiler can see. - -Objective-C ``(copy)`` properties should behave closely enough to Swift ``val`` -properties to be able to vend Objective-C ``(copy)`` properties to Swift as -``val`` properties, and vice versa. - -Objective-C protocols ---------------------- - -In Objective-C, only classes can conform to protocols, and the ``This`` type -is thus presumed to have references semantics. Swift protocols -imported from Objective-C or declared as ``[objc]`` could be conformed to by -``val`` types, but doing so would need to incur an implicit copy to the heap -to create a ``ref`` value to conform to the protocol. - -How This Design Improves Swift -============================== - -1. You can choose semantics at the point of use. The designer of a - type doesn't know whether you will want to use it via a reference; - she can only guess. You might *want* to share a reference to a - struct, tuple, etc. You might *want* some class type to be a - component of the value of some other type. We allow that, without - requiring awkward explicit wrapping, and without discarding the - obvious defaults for types that have them. - -2. We provide a continuum of strictness in which to program. If - you're writing a script, you can go with ``var`` everywhere: don't - worry; be happy. If you're writing a large-scale program and want - to be very sure of what you're getting, you can forbid ``var`` - except in carefully-vetted generic functions. The choice is yours. - -3. We allow generic programmers to avoid subtle semantic errors by - explicitly specifying value or reference semantics where it - matters. - -4. We move the cases where values and references interact much closer - to, and arguably into, the “easy” zone. - -How This Design Beats Rust/C++/C#/etc. -====================================== - -* Simple programs stay simple. Rust has a great low-level memory safety - story, but it comes at the expense of ease-of-use. You can't learn - to use that system effectively without confronting three `kinds`__ - of pointer, `named lifetimes`__, `borrowing managed boxes and - rooting`__, etc. By contrast, there's a path to learning swift that - postpones the ``val``\ /``ref`` distinction, and that's pretty much - *all* one must learn to have a complete understanding of the object - model in the “easy” and “safe” zones. - -__ http://static.rust-lang.org/doc/tutorial.html#boxes-and-pointers -__ http://static.rust-lang.org/doc/tutorial-borrowed-ptr.html#named-lifetimes -__ http://static.rust-lang.org/doc/tutorial-borrowed-ptr.html#borrowing-managed-boxes-and-rooting - -* Simple programs stay safe. C++ offers great control over - everything, but the sharp edges are always exposed. This design - allows programmers to accomplish most of what people want to with - C++, but to do it safely and expressively. As with the rest of Swift, - the sharp edges are still available as an opt-in feature, and - without harming the rest of the language. - -* Unlike C++, types meant to be reference types, supporting - inheritance, aren't copyable by default. This prevents inadvertent - slicing and wrong semantics. - -* By retaining the ``class`` vs. ``struct`` distinction, we give type - authors the ability to provide a default semantics for their types - and avoid confronting their users with a constant ``T*`` vs. ``T`` - choice like C/C++. - -* C# also provides a ``class`` vs. ``struct`` distinction with a - generics system, but it provides no facilities for nontrivial value semantics - on struct types, and the only means for writing generic - algorithms that rely on value or reference semantics is to apply a - blunt ``struct`` or ``class`` constraint to type parameters and limit the - type domain of the generic. By generalizing both value and reference semantics - to all types, we allow both for structs with interesting value semantics and - for generics that can reliably specify and use value or reference semantics - without limiting the types they can be used with. - -``structs`` Really Should Have Value Semantics -============================================== - -It is *possible* to build a struct with reference semantics. For -example, - -..parsed-literal:: - - struct XPair - { - constructor() { - // These Xs are notionally **part of my value** - first = new X - second = new X - } - **ref** first : X - **ref** second : X - } - -However, the results can be surprising: - -.. parsed-literal:: - - **val** a : XPair // I want an **independent value**, please! - val b = a // and a copy of that value - a.first.mutate() // Oops, changes b.first! - -If ``XPair`` had been declared a class, :: - - val a : XPair // I want an independent value, please! - -would only compile if ``XPair`` was also ``Clonable``, thereby -protecting the user's intention to create an independent value - - -Getting the ``ref`` out of a ``class`` instance declared ``val`` -================================================================ - -A ``class`` instance is always accessed through a reference, but when -an instance is declared ``val``, that reference is effectively hidden -behind the ``val`` wrapper. However, because ``this`` is passed to -``class`` methods as a reference, we can unwrap the underlying ``ref`` -as follows:: - - val x : SomeClass - - extension SomeClass { - func get_ref() { return this } - } - - ref y : x.get_ref() - y.mutate() // mutates x - -Teachability -============ - -By expanding the type system we have added complexity to the language. -To what degree will these changes make Swift harder to learn? - -We believe the costs can be mitigated by teaching plain ``var`` -programming first. The need to confront ``val`` and ``ref`` can be -postponed until the point where students must see them in the -interfaces of library functions. All the same standard library -interfaces that could be expressed before the introduction of ``val`` -and ``ref`` can still be expressed without them, so this discovery can -happen arbitrarily late in the game. However, it's important to -realize that having ``val`` and ``ref`` available will probably change -the optimal way to express the standard library APIs, and choosing -where to use the new capabilities may be an interesting balancing act. - -(Im)Mutability -============== - -We have looked, but so far, we don't think this proposal closes (or, -for that matter, opens) the door to anything fundamentally new with -respect to declared (im)mutability. The issues that arise with -explicit ``val`` and ``ref`` also arise without them. - -Bikeshed -======== - -There are a number of naming issues we might want to discuss. For -example: - -* ``var`` is only one character different from ``val``. Is that too - confusable? Syntax highlighting can help, but it might not be enough. - - * What about ``let`` as a replacement for ``var``? - There's always the dreaded ``auto``. - - * Should we drop ``let``\ /``var``\ /``auto`` for ivars, because it - “just feels wrong” there? - -* ``ref`` is spelled like ``[inout]``, but they mean very different things - - * We don't think they can be collapsed into one keyword: ``ref`` - requires shared ownership and is escapable and aliasable, unlike - ``[inout]``. - - * Should we spell ``[inout]`` differently? I think at a high level - it means something like “``[rebind]`` the name to a new value.” - -* Do we want to consider replacing ``struct`` and/or ``class`` with - new names such as ``valtype`` and ``reftype``? We don't love those - particular suggestions. One argument in favor of a change: - ``struct`` comes with a strong connotation of weakness or - second-class-ness for some people. diff --git a/docs/weak.md b/docs/weak.md new file mode 100644 index 0000000000000..e1ff7bcfe1a98 --- /dev/null +++ b/docs/weak.md @@ -0,0 +1,1097 @@ +Weak References +=============== + +Author + +: John McCall + +Date + +: 2013-05-23 + +**Abstract:** This paper discusses the general concept of weak +references, including various designs in other languages, and proposes +several new core language features and a more sophisticated runtime +support feature which can be exploited in the standard library. + +> **warning** +> +> This document was used in planning Swift 1.0; it has not been kept + +> up to date and does not describe the current or planned behavior of +> Swift. + +Reference Graphs +---------------- + +Let's run through some basic terminology. + +Program memory may be seen abstractly as a (not necessarily connected) +directed graph where the nodes are objects (treating transient +allocations like local scopes as objects) and the edges are references. +(Multi-edges and self-edges are permitted, of course.) + +We may assign a level of strength to each of these edges. We will call +the highest level *strong*; all others are some flavor of *weak*. + +Every object has a *strength of reference* which arises from the +reference graph. This can be any level of strength or the special value +*unreferenced*. The strength of a path is the minimum strength of any +edge in the path. The strength of a set of paths is the maximum strength +of all the paths, unless the set is empty, in which case it is +*unreferenced*. + +In general, the implementation is only outright forbidden to deallocate +an object if it is strongly referenced. However, being somehow weakly +referenced may trigger some sort of additional guarantee; see the +Language Precedents section. + +In a cycle-collecting environment, certain nodes are given special +treatment as *roots*; these nodes are always strongly referenced. +Otherwise, the strength of reference for an object is the strength of +all paths from any root to the object. + +In a non-cycle-collecting environment, the strength of reference for an +object is the strength of all the direct references to that object, +taken as length=1 paths. Note that this environmental consideration +becomes a language guarantee: even if the implementation can trivially +prove that an object is referenced only by itself, it is still not +permitted to deallocate the object. + +It is common for certain kinds of reference to not receive a full +guarantee. For example, a strong reference from a local variable may +lose effectiveness as soon as the variable is no longer needed (but +before it formally leaves scope). This is pervasive in GC environments +but also true in e.g. ObjC ARC. + +In some programs, especially event-driven programs like UIs or servers, +it can be useful to consider the *static* reference graph as captured +during a notional steady state. Local scopes may form many references to +objects in the static graph, but unless the scope persists indefinitely +or a major restructuring is triggered, these references are likely to +have no effect on the reference graph, and so their strength of +reference has no semantic importance. Understanding this helps to +explain why many environments provide no direct way to form a local weak +reference. + +Language and Library Precedents +------------------------------- + +We're only going to discuss *managed* precedents here. + +It is possible to create a kind of weak reference by just not providing +any special behavior to an object reference; if the object is +deallocated, the reference will dangle and uses of it will likely crash +or cause corruption. This could happen by e.g. declining to insert +reference-count manipulations for a particular variable (in an ARC-like +environment) or not mapping a variable's frame offset as a pointer in a +type map (in a precise-GC environment). We will call this *dangling +weak*. + +### Objective-C + +All modes of Objective-C automate memory management for synthesized +properties. In GC and ARC, accessors just use the normal semantics for +the underlying ivar (plus copy/atomic guarantees, of course). In MRC, +accessors use dangling weak semantics unless they're +`retain`{.sourceCode} or `copy`{.sourceCode}, in which case they +maintain a +1 refcount invariant on the referent. + +In GC and ARC, variables qualified with `__weak`{.sourceCode} are +immediately zeroed out when the referenced object begins deallocation. +There is no syntactic difference on use; it's just possible that the +value read will be `nil`{.sourceCode} instead of whatever was last +written there, possibly causing the loading code to crash (e.g. on an +ivar access) or silently do nothing (e.g. on a message send). There is +an opt-in warning in ARC for certain uses of values loaded from +`__weak`{.sourceCode} ivars. + +In GC, it is also possible to construct a dangling weak reference by +storing an object pointer into (1) unscanned heap memory or (2) an +instance variable that's not of object-pointer type and isn't qualified +with `__strong`{.sourceCode} or `__weak`{.sourceCode}. Otherwise, object +references are strong (including all references from local scopes). + +In ARC, it is possible to construct a dangling weak reference by using +the `__unsafe_unretained`{.sourceCode} qualifier or by bridging a +pointer value to a C pointer type. + +### C++ + +C++ smart pointers (e.g. `std::unique_ptr`{.sourceCode}) typically +permit the creation of a dangling-weak reference by providing an +accessor to get the pointer as a normal C pointer. (Even if they didn't +have `get()`{.sourceCode}, you could manually call +`operator->`{.sourceCode} to get the same effect.) + +C++'s `std::shared_ptr`{.sourceCode} permits the formation of weak +pointers (`std::weak_ptr`{.sourceCode}) from shared pointers. It is not +possibly to directly use a weak pointer; it must first be converted back +to a `shared_ptr`{.sourceCode}, either by using the +`lock()`{.sourceCode} operation (which produces a null pointer if the +referent has been deallocated) or by directly constructing a +`shared_ptr`{.sourceCode} with the `weak_ptr`{.sourceCode} (which throws +an exception if the referent has been deallocated). There is also a way +to explicitly query whether a `weak_ptr`{.sourceCode} is still valid, +which may be more efficient than checking the result of the cast. + +### Java + +Java does not provide any facility for dangling weak references. The +standard library does provide three levels of weak reference (in +`java.lang.ref`{.sourceCode}). References cannot be re-seated (although +they can be explicitly cleared), and users must call +`get()`{.sourceCode} in order to access the value, which may yield +`null`{.sourceCode}. + +There is a great deal of interesting discussion of these reference +classes [here](http://www.kdgregory.com/index.php?page=java.refobj). + +Java `Reference`{.sourceCode} objects may be constructed with an +optional `ReferenceQueue`{.sourceCode}; if so, then when the object's +reachability changes, the reference object will be added to that queue. +This permits data structures to clean up after cleared soft references +without needing to either periodically scan the entire structure or be +fully lazy. Additional data may be added to the reference object by +subclassing it. + +The references are presented in order of decreasing strength. + +`SoftReference`{.sourceCode} is a sort of quasi-strong reference which +holds onto the object until the VM begins to run out of memory. Soft +references to softly-referenced objects are guaranteed to have been +cleared before the VM can throw an `OutOfMemoryError`{.sourceCode}. The +reference will be cleared before it is added to its reference queue (and +so the reference queue cannot resurrect the object). The intent of soft +references is to enable memory-sensitive caches, but in practice a +memory-sensitive cache would probably want to implement a more subtle +replacement strategy than "drop things at random as soon as memory runs +low". A more interesting use is a memory-guided circuit-breaker: when +building up a very large structure, hold it in a soft reference, and if +that references goes null during construction, just bail out. But that's +a pretty tricky use-case to get right. + +`WeakReference`{.sourceCode} is intended for use in non-memory-sensitive +weak caches, like a uniquing cache; it persists only as long as the +referent is more strongly referenced. The reference will be cleared +before it is added to its reference queue (and so the reference queue +cannot resurrect the object). + +`PhantomReference`{.sourceCode} provides a way to attach extra +finalization to an object without actually using finalizers (which have +several problems, including the ability to resurrect the object). The +phantom reference *always* presents `null`{.sourceCode} as its value and +is therefore itself useless as a reference. Phantom references are +enqueued after the object is finalized and therefore at a point when +there can be no references to the object within the VM at all. However, +the object itself cannot be deallocated until the phantom references are +all cleared or themselves deallocated, which I believe is for the +convenience of native code that may hold a dangling weak reference to +the referent (or which may be able to directly read the reference). + +### .NET + +The `WeakReference`{.sourceCode} class in .NET is similar to Java's +`WeakReference`{.sourceCode} class in that the value cannot be accessed +directly; it must be accessed via the `Target`{.sourceCode} property, +which may yield `null`{.sourceCode}. The reference may be reseated to a +different value. + +Weak references may be created *long*, which permits the target object +to be finalized but not actually deallocated. + +### Python + +A `weakref`{.sourceCode} acts like a function object; it is created with +a particular value, which cannot be reseated. The function will yield +`None`{.sourceCode} if the referent is collected. + +There is library functionality to automatically proxy a value as a weak +reference. An exception is thrown if an operation is performed on the +proxy but the referent has been collected. + +A `weakref`{.sourceCode} may be constructed with a callback function. +The callback will be called after the weak reference is cleared; it is, +however, passed the weak ref object itself. + +### Ruby + +A `WeakRef`{.sourceCode} is automatically a proxy for an object. There +is a `weakref_alive`{.sourceCode} method to query whether the reference +is still alive; another other operation will cause an exception to be +thrown. + +### Rust + +As far as I can tell, there is nothing like a weak reference in Rust at +the moment. + +A *managed pointer* (`@int`{.sourceCode}) is a strong reference subject +to GC. + +An *owning pointer* (`~int`{.sourceCode}) is a strong reference that +cannot be cloned (copying the pointer actually copies the underlying +data). + +A *borrowed pointer* (`&int`{.sourceCode}) is essentially a dangling +weak reference that is subject to static restrictions which ensure that +it doesn't actually dangle. It is thus primarily a performance +optimization. + +A *raw pointer* (`*int`{.sourceCode}) is a dangling weak reference. + +### Haskell + +Yes, of course Haskell has weak references. + +A `Weak t`{.sourceCode} is an association between a hidden key and a +visible value of type `t`{.sourceCode}. `doRefWeak theRef`{.sourceCode} +is an `IO (Maybe t)`{.sourceCode}. + +A weak reference may be constructed with an optional +`IO ()`{.sourceCode} which will be run when the referent is collected. +This finalizer may (somehow) refer to the key and value without itself +keeping them alive; it is also explicitly permitted to resurrect them. + +Use Cases +--------- + +There are many problems that are potentially addressable with +functionality like weak references. It is not at all obvious that they +should be addressed with the same language feature. + +### Back references + +Given that Swift is not cycle-collecting, far and away the most +important use case in the static reference graph is that of the +*back-reference*: a reference *R* to an object which holds a strong +reference (possibly indirectly) to the object holding *R*. Examples +include: + +- A 'previousNode' pointer in a doubly-linked list. +- A 'parent' pointer in a render tree. +- An edge in a general graph structure. + +These have several properties in common: + +- Using strong references would require a lot of explicit code to tear + down the reference cycles. +- These references may be accessed very frequently, so performance + is important. +- It is not always feasible to make these references valid immediately + on construction. +- Traversing a reference after the referent is deallocated is likely a + sign that something has been kept alive longer than it was meant + to be. However, programmers may reasonably differ about the correct + response to this: crashing, and therefore encouraging the programmer + to track down a root cause, or simply writing the operation to + handle both cases correctly. Ultimately, this choice comes down + to philosophy. + +### Caches + +Weak caches are used in order to prevent a cache from taking over all +available memory. By being tied to the reachability of a value, the +cache prevents entries from spuriously expiring when their values are +still in active use; but by using weak references, the cache permits the +system to deallocate values that are no longer in use. + +Generally, a data structure using weak references extensively also needs +some way to receive notification that the weak reference was collected. +This is because entries in the data structure are likely to have +significant overhead even if the value is collected. A weak data +structure which receives no notification that a reference has been +invalidated must either allow these entries to accumulate indefinitely +or must periodically scan the entire structure looking for stale +entries. + +A weak reference which permits immediate deallocation of its referent +when the last strong reference is dropped is substantially less useful +for the implementation of a weak cache. It is a common access pattern +(for, say, a memoizing cache) for a value to be looked up many times in +rapid succession, but for each use to be temporarily disjoint from the +others. A naive use of weak references in this case will simply cause +the cache to thrash. This problem is less likely to arise in an +environment with nondeterministic collection because the entry is likely +to service multiple lookups between collections. + +It is likely that users implementing weak data structures would prefer a +highly flexible infrastructure centered around resurrection and +notifications of reaching a zero refcount than a more rigid system built +directly into the language. Since the Swift model is built around +statically-inserted operations rather than a memory scanner, this is +much more workable. + +### External Finalization + +Finalization models built around calling a method on the finalized +object (such as Objective-C's `-dealloc`{.sourceCode}) suffer from a +number of limitations and problems: + +> - Since the method receives a pointer to the object being +> deallocated, the implementation must guard against attempts to +> resurrect the object. This may complicate and/or slow down the +> system's basic reference-management logic, which tends to be quite +> important for performance. +> - Since the method receives a pointer to the object being +> deallocated, the implementation must leave the object at least a +> minimally valid state until the user code is complete. For +> example, the instance variables of a subclass cannot be destroyed +> until a later phase of destruction, because a superclass finalizer +> might invoke subclass behavior. (This assumes that the dynamic +> type of the object does not change during destruction, which is an +> alternative that brings its own problems.) +> - Finalization code must be inherent to the object; other objects +> cannot request that code be run when the object is deallocated. +> For example, an object that registers itself to observe a certain +> event source must explicitly deregister itself in a finalizer; the +> event source cannot simply automatically drop the object when it +> is deallocated. + +### Optimization + +Functions often create a large number of temporary references. In a +reference-counting environment like Swift, these references require the +implementation to implicitly perform operations to increment and +decrement the reference count. These operations can be quite fast, but +they are not free, and our experience has been that the accumulated cost +can be quite significant. A straightforward local static analysis can +eliminate many operations, but many others will be blocked by +abstraction barriers, chiefly dynamically-dispatched calls. Therefore, +if Swift is to allow precise performance control, it is important to be +able to allow motivated users to selectively control the emission of +reference-counting operations. + +This sort of control necessarily permits the creation of dangling weak +references and so is not safe. + +Proposal Overview +----------------- + +Looking at these use-cases, there are two main thrusts: + +> - There is a general need to set up back references to objects. +> These references must be designed for convenient use by +> non-expert users. +> - There are a number of more sophisticated use cases which require +> notification or interruption of deallocation; these can be used in +> the implementation of higher-level abstractions like weak caches. +> Here it is reasonable to expect more user expertise, such that +> power and flexibility should take priority over ease of use. + +The second set of use cases should addressed by library types working on +top of basic runtime support. + +The first set of use cases will require more direct language support. To +that end, I propose adding two new variable attributes, +`@weak`{.sourceCode} and `@unowned`{.sourceCode}. I also propose a small +slate of new features which directly address the problem of capturing a +value in a closure with a different strength of reference than it had in +the enclosing context. + +Proposed Variable Attributes +---------------------------- + +In the following discussion, a "variable-like" declaration is any +declaration which binds a name to a (possibly mutable) value of +arbitrary type. Currently this is just `var`{.sourceCode}, but this +proposal also adds `capture`{.sourceCode}, and we may later add more +variants, such as `const`{.sourceCode} or `val`{.sourceCode} or the +like. + +### `@weak`{.sourceCode} + +`weak`{.sourceCode} is an attribute which may be applied to any +variable-like declaration of reference type `T`{.sourceCode}. For +type-system purposes, the variables behaves like a normal variable of +type `Optional`{.sourceCode}, except: + +> - it does not maintain a +1 reference count invariant and +> - loading from the variable after the current referent (if present) +> has started destruction will result in a `Nothing`{.sourceCode} +> value, indistinguishable from the normal case. + +The semantics are quite similar to weak references in other environments +(particularly Objective-C) except that the change in formal type forces +the user of the value to check its validity before using it. + +It doesn't really matter how the compiler would find the type +`Optional`{.sourceCode}; compiler-plus-stdlib magic, most likely. I +do not think the type should be `Weak`{.sourceCode} because that +would effectively make this a type attribute and thus make it too easy +to accidentally preserve the value as a weak reference. See the section +discussing type attributes vs. declaration attributes. + +Giving the variable a consistent type of `Optional`{.sourceCode} +permits the user to assign `Nothing`{.sourceCode} into it and therefore +removes the somewhat odd expressive possibility of a reference that can +only be missing if the object has been deallocated. I think this is an +acceptable cost of making a cleaner feature. + +One alternative to using `Optional`{.sourceCode} would be to simply +treat the load as a potentially-failing operation, subject to the (not +yet precisely designed) language rules for error handling. This would +require the runtime to potentially synthesize an error event, which +could then propagate all the way to the end-user if the programmer +accidentally failed to check the result in a context that permitted +error propagation outwards. That's bad. + +A slightly different alternative would be to treat it as a form of error +orthogonal to the standard user-error propagation. This would be cleaner +than changing the type of the variable but can't really be designed +without first having a solid error-handling design. + +### `@unowned`{.sourceCode} + +`unowned`{.sourceCode} is an attribute which may be applied to any +"variable-like" declaration of reference type `T`{.sourceCode}. For +type-system purposes, the variable behaves exactly like a normal +variable of type `T`{.sourceCode}, except: + +> - it does not maintain a +1 reference count invariant and +> - loading from the variable after the referent has started +> destruction causes an assertion failure. + +This is a refinement of `weak`{.sourceCode} focused more narrowly on the +case of a back reference with relatively tight validity invariants. This +is also the case that's potentially optimizable to use dangling weak +references; see below. + +This name isn't really optimal. We've considered several different +candidates: + +> - `weak`{.sourceCode} is a poor choice because our semantics are +> very different from weak references in other environments where +> it's valid to access a cleared reference. Plus, we need to expose +> those semantics, so the name is claimed. +> - `backref`{.sourceCode} is strongly evocative of the major use case +> in the static reference graph; this would encourage users to use +> it for back references and to consider alternatives for other +> cases, both of which I like. The latter also makes the +> husk-leaking implementation (see below) more palatable. It also +> contrasts very well with `weak`{.sourceCode}. However, its +> evocativeness makes it unwieldy to use for local +> reference-counting optimizations. +> - `dangling`{.sourceCode} is more general than +> `backref`{.sourceCode}, but it has such strong negative +> associations that it wouldn't be unreasonable for users to assume +> that it's unsafe (with all the pursuant debugging difficulties) +> based on the name alone. I don't think we want to discourage a +> feature that can help users build tighter invariants on +> their classes. +> - `unowned`{.sourceCode} is somewhat cleaner-looking, and it isn't +> as tied to a specific use case, but it does not contrast with +> `weak`{.sourceCode} *at all*; only someone with considerable +> exposure to weak references would understand why we named each one +> the way we did, and even they are likely to roll their eyes at us. +> But it's okay for a working proposal. + +#### Asserting and Uncheckable + +There should not be a way to check whether a `unowned`{.sourceCode} +reference is still valid. + +- An invalid back-reference is a consistency error that we should + encourage programmers to fix rather than work around by spot-testing + for validity. +- Contrariwise, a weak reference that might reasonably be invalidated + during active use should be checked for validity at *every* use. We + can provide a simple library facility for this pattern. +- Permitting implicit operations like loads to fail in a recoverable + way may end up complicating the language model for error-handling. +- By disallowing recovery, we create a model where the only need to + actually register the weak reference with the system is to enable a + consistency check. Users who are confident in the correctness of + their program may therefore simply disable the consistency check + without affecting the semantics of the program. In this case, that + leaves the variable a simple dangling-weak reference. + +#### Implementation + +The standard implementation for a weak reference requires the address of +the reference to be registered with the system so that it can be cleared +when the referent is finalized. This has two problems: + +- It forces the runtime to maintain a side-table mapping objects to + the list of weak references; generally this adds an allocation per + weakly-referenced object. +- It forces the representation of weak references to either be + non-address-invariant or to introduce an extra level of indirection. + +For some use cases, this may be warranted; for example, in a weak cache +it might come out in the noise. But for a simple back-reference, these +are substantial penalties. + +Dave Z. has proposed instead using a weak refcount, analogous to a +strong refcount. Ownership of a weak retain can be easily transferred +between locations, and it does not require a side-table of an object's +weak references. However, it does have a very important downside: since +the system cannot clear all the references, it is impossible to actually +deallocate an object that is still weakly-referenced (although it can be +finalized). Instead, the system must wait for all the weak references to +at least be accessed. We call this "husk leaking". + +This downside could be a problem for a general weak reference. However, +it's fine for a back-reference, which we expect to typically be +short-lived after its referent is finalized. + +### Declaration Attribute or Type Attribute + +This proposal describes `weak`{.sourceCode} and `unowned`{.sourceCode} +as declaration attributes, not type attributes. + +As declaration attributes, `@unowned`{.sourceCode} and +`weak`{.sourceCode} would be permitted on any `var`{.sourceCode} +declaration of reference type. Their special semantics would be a +property only of the declaration; in particular, they would not change +the type (beyond the shift to `Optional`{.sourceCode} for +`weak`{.sourceCode}) , and more generally the type-checker would not +need to know about them. The implementation would simply use different +behavior when loading or storing that variable. + +As a type attribute, `weak`{.sourceCode} and `@unowned`{.sourceCode} +would be permitted to appear at arbitrary nested locations in the type +system, such as tuple elements, function result types, or generic +arguments. It would be possible to have both lvalues and rvalues of +so-qualified type, and the type checker would need to introduce implicit +conversions in the right places. + +These implicit conversions require some thought. Consider code like the +following: + + func moveToWindow(newWindow : Window) { + var oldWindow = self.window // an @unowned back reference + oldWindow.hide() // might remove the UI's strong reference + oldWindow.remove(self) + newWindow.add(self) + } + +It would be very unfortunate if the back-reference nature of the +`window`{.sourceCode} property were somehow inherited by +`oldWindow`{.sourceCode}! Something, be it a general rule on loading +back-references or a type-inference rule, must introduce an implicit +conversion and cause the `unowned`{.sourceCode} qualifier to be +stripped. + +That rule, however, is problematic for generics. A key goal of generics +is substitutability: the semantics of generic code should match the +semantics of the code you'd get from copy-pasting the generic code and +substituting the arguments wherever they're written. But if a generic +argument can be `@unowned T`{.sourceCode}, then this won't be true; +consider: + + func foo(x : T) { + var y = x + ... + } + +In substituted code, `y`{.sourceCode} would have the qualifier stripped +and become a strong reference. But the generic type-checker cannot +statically recognize that this type is `unowned`{.sourceCode}-qualified, +so in order to match semantics, this decision must be deferred until +runtime, and the type-checker must track the unqualified variant of +`T`{.sourceCode}. This adds a great deal of complexity, both to the +implementation and to the user model, and removes many static +optimization opportunities due to potential mismatches of types. + +An alternative rule would be to apply an implicit "decay" to a strong +reference only when a type is known to be a `unowned`{.sourceCode} type. +It could be argued that breaking substitution is not a big deal because +other language features, like overloading, can already break it. But an +overlapping overload set with divergent behavior is a poor design which +itself violates substitution, whereas this would be a major unexcused +deviation. Furthermore, preserving the weakness of a reference is not a +safe default, because it permits the object to be destroyed while a +reference is still outstanding. + +In addition, any implementation model which permits the safety checks on +`unowned`{.sourceCode}s to be disabled will require all code to agree +about whether or not the checks are enabled. When this information is +tied to a declaration, this is easy, because declarations are ultimately +owned by a particular component, and we can ask how that component is +compiled. (And we can demand that external APIs commit to one level of +safety or the other before publishing.) The setting for a type, on the +other hand, would have to be determined by the module which "wrote the +type", but again this introduces a great deal of complexity which one +can only imagine settling on the head of some very confused user. + +For all these reasons, I feel that making `weak`{.sourceCode} and +`unowned`{.sourceCode} type attributes would be unworkable. However, +there are still costs to making them declaration attributes: + +- It forces users to use awkward workarounds if they want to make, + say, arrays of back-references. +- It makes back-references less composable by, say, preventing them + from being stored in a tuple. +- It complicates SIL and IR-gen by making the rules for manipulating a + physical variable depend on more than just the type of the variable. +- It automatically enables certain things (like passing the address of + a `@unowned`{.sourceCode} variable of type `T`{.sourceCode} to a + `inout T`{.sourceCode} parameter) that perhaps ought to be more + carefully considered. + +The first two points can be partly compensated for by adding library +types to wrap a back-reference. Accessing a wrapped reference will +require extra syntax, and it runs the same risk of accidentally +preserving the weakness of a reference that I discussed above. However, +note that these problems are actually at odds: requiring extra syntax to +access the wrapped reference will leave breadcrumbs making it clear when +the change-over occurs. For more on this, see the library section. + +### `weak`{.sourceCode}-Capable Types + +Swift reference types can naturally be made to support any kind of +semantics, and I'm taking it on faith that we could enhance ObjC objects +to support whatever extended semantics we want. There are, however, +certain Swift value types which have reference-like semantics that it +could be useful to extend `weak`{.sourceCode} (and/or +`unowned`{.sourceCode}) to: + +- Being able to conveniently form an optional back-reference seems + like a core requirement. If `weak`{.sourceCode} were a type + attribute, we could just expect users to write + `Optional<@weak T>`{.sourceCode}; as a declaration attribute, this + is substantially more difficult. I expect this to be common enough + that it'll be unreasonable to ask users to use + `Optional>`{.sourceCode}. +- Being able to form a back-reference to a slice or a string seems + substantially less important. + +One complication with extending `weak`{.sourceCode} to value types is +that generally the implementing type will need to be different from the +underlying value type. Probably the best solution would be to hide the +use of the implementing type from the type system outside of the +well-formedness checks for the variable; SIL-gen would lower the field +to its implementing type using the appropriate protocol conformances. + +As long as we have convenient optional back-references, though, we can +avoid designing a general feature for 1.0. + +### Generic Weak Support + +All other uses for weak references can be glossed as desiring some +amount of additional work to occur when the strong reference count for +an object reaches zero. This necessarily entails a global side-table of +such operations, but I believe that's acceptable as long as it's +relegated to less common use-cases. + +It is important that the notification mechanism not require executing +code re-entrantly during the finalization process. + +I suggest adopting an interface centered around the Java concept of a +`ReferenceQueue`{.sourceCode}. A reference structure is registered with +the runtime for a particular object with a particular set of flags and +an optional reference queue: + + struct Reference { + void *Referent; // must be non-null upon registration + struct ReferenceQueue *Queue; // must be valid or null + size_t Reserved[2]; + }; + + void swift_registerReference(struct Reference *reference, + size_t flags); + +The user/library code is responsible for allocating these structures and +initializing the first two fields, and it may include arbitrary fields +before or after the `Reference`{.sourceCode} section, but while the +reference is registered with the runtime, the entire +`Reference`{.sourceCode} section becomes reserved and user/library code +must not access it in any way. + +The flags include: + +- A priority. Should be constrained to two or three bits. References + are processed in order of decreasing priority; as long as a + reference still exists with higher priority, references with lower + priority cannot be processed. Furthermore, as long as any reference + exists, the referent cannot be finalized. +- Whether to automatically clear the reference when processing it. + Note that a cleared reference is still considered to be registered + with the runtime. + +These could be combined so that e.g. even priorities cause an automatic +clear and odd priorities do not; this would avoid some odd effects. + +The runtime may assume that explicit user operations on the same +reference will not race with each other. However, user operations on +different references to the same referent may be concurrent, either with +each other or with other refcount operations on the referent. + +The operations on references are as follows: + + void *swift_readReference(struct Reference *reference); + +This operation atomically either produces a strong reference to the +referent of the given object or yields `null`{.sourceCode} if the +referent has been finalized (or if the referent is `null`{.sourceCode}). +The reference must currently be registered with the runtime: + + void swift_writeReference(struct Reference *reference, + void *newReferent); + +This operation changes the referent of the reference to a new object, +potentially `null`{.sourceCode}. The argument is taken at +0. The +reference must currently be registered with the runtime. The reference +keeps the same flags and reference queue: + + void swift_unregisterReference(struct Reference *Reference); + +This operation clears a reference, removes it from its reference queue +(if it is enqueued), and unregisters it from the runtime. The reference +must currently be registered with the runtime. + +I propose the following simple interface to a ReferenceQueue; arguably, +however, it ought to be a reference-counted library type with a small +amount of native implementation: + + struct ReferenceQueue; + struct ReferenceQueue *swift_createReferenceQueue(void); + +Allocate a new reference queue: + + void swift_destroyReferenceQueue(struct ReferenceQueue *queue); + +Destroy a reference queue. There must not be any references with this +queue currently registered with the runtime: + + struct Reference *swift_pollReferenceQueue(struct ReferenceQueue *queue); + +Proposed Rules for Captures within Closures +------------------------------------------- + +When a variable from an enclosing context is referenced in a closure, by +default it is captured by reference. Semantically, this means that the +variable and the enclosing context(s) will see the variable as a single, +independent entity; modifications will be seen in all places. In terms +of the reference graph, each context holds a strong reference to the +variable itself. (In practice, most local variables captured by +reference will not require individual allocation; usually they will be +allocated as part of the closure object. But in the formalism, they are +independent objects.) + +Closures therefore make it fairly easy to introduce reference cycles: +for example, an instance method might install a closure as an event +listener on a child object, and if that closure refers to +`self`{.sourceCode}, a reference cycle will be formed. This is an +indisputable drawback of an environment which cannot collect reference +cycles. + +Relatively few languages both support closures and use +reference-counting. I'm not aware of any that attempt a language +solution to the problem; the usual solution is to the change the +captured variable to use weak-reference semantics, usually by copying +the original into a new variable used only for this purpose. This is +annoyingly verbose, clutters up the enclosing scope, and duplicates +information across multiple variables. It's also error-prone: since both +names are in scope, it's easy to accidentally refer to the wrong one, +and explicit capture lists only help if you're willing to be very +explicit. + +A better alternative (which we should implement in Objective-C as well) +is to permit closures to explicitly specify the semantics under which a +variable is captured. In small closures, it makes sense to put this near +the variable reference; in larger closures, this can become laborious +and redundant, and a different mechanism is called for. + +In the following discussion, a *var-or-member expression* is an +expression which is semantically constrained to be one of: + +> - A reference to a local variable-like declaration from an +> enclosing context. +> - A member access thereof, possibly recursively. + +Such expressions have two useful traits: + +> - They always end in an identifier which on some level meaningfully +> identifies the object. +> - Evaluating them is relatively likely (but not guaranteed) to not +> have interesting side effects, and so we feel less bad about +> apparently shifting their evaluation around. + +### Decorated Capture References + +Small closures are just as likely to participate in a reference cycle, +but they suffer much more from extraneous syntax because they're more +likely to be center-embedded in interesting expressions. So if there's +anything to optimize for in the grammar, it's this. + +I propose this fairly obvious syntax: + + button1.setAction { unowned(self).tapOut() } + button2.setAction { if (weak(self)) { weak(self).swapIn() } } + +The operand is evaluated at the time of closure formation. Since these +references can be a long way from the top of the closure, we don't want +to allow a truly arbitrary expression here, because the order of +side-effects in the surrounding context could be very confusing. So we +require the operand to be an `expr-var-or-member`{.sourceCode}. More +complicated expressions really ought to be hoisted out to a separate +variable for legibility anyway. + +I do believe that being able to capture the value of a property +(particularly of `self`{.sourceCode}) is very important. In fact, it's +important independent of weak references. It is often possible to avoid +a reference cycle by simply capturing a specific property value instead +of the base object. Capturing by value is also an expressivity +improvement: the programmer can easily choose between working with the +property's instantaneous value (at the time the closure is created) or +its current value (at the time the closure is invoked). + +Therefore I also suggest a closely-related form of decoration: + + button3.setAction { capture(self.model).addProfitStep() } + +This syntax would force `capture`{.sourceCode} to become a real keyword. + +All of these kinds of decorated references are equivalent to adding a +so-attributed `capture`{.sourceCode} declaration (see below) with a +nonce identifier to the top of the closure: + + button1.setAction { + capture @unowned _V1 = self + _V1.tapOut() + } + button2.setAction { + capture @weak _V2 = self + if (_V2) { _V2.swapIn() } + } + button3.setAction { + capture _V3 = self.model + _V3.addProfitStep() + } + +If the operand of a decorated capture is a local variable, then that +variable must not be the subject of an explicit `capture`{.sourceCode} +declaration, and all references to that variable within the closure must +be identically decorated. + +The requirement to decorate all references can add redundancy, but only +if the programmer insists on decorating references instead of adding an +explicit `capture`{.sourceCode} declaration. Meanwhile, that redundancy +aids both maintainers (by preventing refactors from accidentally +removing the controlling decoration) and readers (who would otherwise +need to search the entire closure to understand how the variable is +captured). + +Captures with identical forms (the same sequence of members of the same +variable) are merged to the same capture declaration. This permits type +refinement to work as expected, as seen above with the +`weak`{.sourceCode} capture. It also guarantees the elimination of some +redundant computation, such as with the `state`{.sourceCode} property in +this example: + + resetButton.setAction { + log("resetting state to " + capture(self.state)) + capture(self.model).setState(capture(self.state)) + } + +I don't see any immediate need for other kinds of capture decoration. In +particular, I think back references are likely to be the right kind of +weak reference here for basically every use, and I don't think that +making it easy to capture a value with, say, a zeroable weak reference +is important. This is just an intuition deepened by hallway discussions +and close examination of a great many test cases, so I concede it may +prove to be misguided, in which case it should be easy enough to add a +new kind of decoration (if we're willing to burn a keyword on it). + +### `capture`{.sourceCode} Declarations + +This feature generalizes the above, removing some redundancy in large +closures and adding a small amount of expressive power. + +A `capture`{.sourceCode} declaration can only appear in a closure: an +anonymous closure expression or `func`{.sourceCode} declaration that +appears directly within a function. (By "directly" I mean not within, +say, a local type declaration within the function). +`capture`{.sourceCode} will need to at least become a context-sensitive +keyword. + +A closure may contain multiple `capture`{.sourceCode} declarations, but +they must all appear at the very top. One reason is that they can affect +the capture semantics within the closure, so someone reading the closure +should be able to find them easily. Another reason is that they can +involve executing interesting code in the enclosing context and so +should reliably appear near the closure formation site in the source +code: + + decl ::= decl-capture + decl-capture ::= 'capture' attribute-list '=' expr-var-or-member + decl-capture ::= 'capture' attribute-list decl-capture-expr-list + decl-capture-expr-list ::= expr-var-or-member + decl-capture-expr-list ::= expr-var-or-member ',' decl-capture-expr-list + +Both forms of `capture`{.sourceCode} declaration cause one or more +fields to be created within the closure object. At the time of creating +the closure, these fields are initialized with the result of evaluating +an expression in the enclosing context. Since the expression is +evaluated in the enclosing context, it cannot refer to "previous" +captures; otherwise we could have some awkward ambiguities. I think we +should reserve the right to not execute an initializer if the closure +will never be called; this is more important for local +`func`{.sourceCode} declarations than for anonymous closure expressions. + +The fields introduced by `capture`{.sourceCode} declarations should be +immutable by default, but programmers should be able to write something +like `capture var ...`{.sourceCode} to make them mutable. Locking down +on mutation isn't quite as important as it is with implicit captures +(where it's easy to accidentally believe you're modifying the enclosing +variable) or even explicit captures in C++11 lambdas (where copies of +the lambda object will copy the capture field and thus produce +mystifying behavior in uncareful code), but it's still a source of easy +mistakes that should require manual intervention to enable. This all +presumes that we eventually design mutability into the language, of +course. + +In the pattern-initializer form, the field names are given explicitly by +the pattern. The abbreviated form borrows the name of the captured +member or local variable. In either case, names should be subject to the +usual shadowing rules, whatever they may be, with the exception that +capturing an enclosing variable with the abbreviated form is not +problematic. + +Attributes on a `capture`{.sourceCode} declaration affect all the fields +it declares. + +Let's spell out some examples. I expect the dominant form to be a simple +identifier: + + capture @unowned foo + +This captures the current value of whatever `foo`{.sourceCode} resolves +to (potentially a member of `self`{.sourceCode}!) and binds it within +the closure as a back-reference. + +Permitting the slightly more general form: + + capture window.title + +allows users to conveniently capture specific values without mucking up +the enclosing scope with tons of variables only needed for setting up +the closure. In particular, this makes it easy to capture specific +fields out of an enclosing `self`{.sourceCode} object instead of +capturing the object itself; that, plus forcing uses of +`self`{.sourceCode} to be explicit in closures, would help users to +conveniently avoid a class of inadvertent retain cycles. + +I've included the general pattern-initializer form mostly for ease of +exposition. It adds no major expressivity improvements over just +creating a variable in the enclosing context. It does avoid cluttering +the enclosing scope with new variables and permits captures to be +locally renamed, and it can very convenient if introducing a new +variable in the enclosing scope would be complicated (for example, if +there were a reason to prefer using a single statement there). I don't +think it does any harm, but I wouldn't mourn it, either. I do think that +generalizing the initializer to an arbitrary expression would be a +serious mistake, because readers are naturally going to overlook code +which occurs inside the closure, and promoting side effects there would +be awful. + +It would be nice to have a way to declare that a closure should not have +any implicit captures. I don't have a proposal for that right now, but +it's not important for weak references. + +### Nested Closures + +It is important to spell out how these rules affect captures in nested +closures. + +Recall that all of the above rules can be transformed into `capture` +declarations at the beginning of a closure, and that `capture` +declarations always introduce a by-value capture instead of a +by-reference capture. + +A by-reference capture is always of either a local variable or a +`capture` declaration. In neither case do we currently permit such +captures to "drag in" other declarations silently, to the extent that +this is detectable. This means that mutable `capture` declarations that +are themselves captured by reference must be separately allocated from +the closure object, much like what happens with normal locals captured +by reference. + +I've considered it quite a bit, and I think that a by-value capture of a +variable from a non-immediately enclosing context must be made +ill-formed. At the very least, it must be ill-formed if either the +original variable is mutable (or anything along the path is, if it +involves properties) or the capture adds `@unowned`. + +This rule will effectively force programmers to use extra variables or +`capture`s as a way of promising validity of the internal captures. + +The motivation for this prohibition is that the intent of such captures +is actually quite ambiguous and/or dangerous. Consider the following +code: + + async { doSomething(); GUI.sync { unowned(view).fireCompleted() } } + +The intent of this code is to have captured a back-reference to the +value of `view`{.sourceCode}, but it could be to do so at either of two +points in time. The language must choose, and in this hypothetical it +must do so without further declaration of intent and without knowledge +of when and how many times the closures will be called. + +Suppose that we capture the value at the earlier point, when we form the +outer closure. This will behave very surprisingly if `view`{.sourceCode} +is in fact mutated; it may be easier to imagine this if, instead of a +simple local variable, it was instead a path like +`self.view`{.sourceCode}. And it's not clear that forming a +back-reference at this earlier point is actually valid; it is easy to +imagine code that would rely on the intermediate closure holding a +strong reference to the view. Furthermore, and crucially, these issues +are inextricable: we cannot somehow keep track of the mutable variable +but only hold its value weakly. + +But suppose instead that we capture the value at the later point. Then +the intermediate closure will capture the `view`{.sourceCode} variable +by reference, which means that in effect it will hold +`view`{.sourceCode} strongly. This may actually completely subvert the +user's desired behavior. + +It's not clear to me right now whether this problem applies equally to +explicit `capture`{.sourceCode} declarations. Somehow decorated +expressions seem more ambiguous in intent, probably because the syntax +is more thoughtlessly convenient. On the other hand, making the +decoration syntax not just a shorthand for the explicit declarations +makes the model more complex, and it may be over-complex already. + +So in summary, it would be best to enforce a strong prohibition against +these dangerous multi-level captures. We can tell users to introduce +secondary variables when they need to do subtle things across several +closure levels. + +Library Directions +------------------ + +The library should definitely provide the following types: + +- `UnownedReference`{.sourceCode}: a fragile value type with a + single public `unowned`{.sourceCode} field of type `T`{.sourceCode}. + There should be an implicit conversion *from* `T`{.sourceCode} so + that obvious initializations and assignments succeed. However, there + should not be an implicit conversion *to* `T`{.sourceCode}; this + would be dangerous because it could hide bugs introduced by the way + that e.g. naive copies into locals will preserve the weakness of + the reference. + + In keeping with our design for `unowned`{.sourceCode}, I think this + type should should actually be an alias to either + `SafeUnownedReference`{.sourceCode} or + `UnsafeUnownedReference`{.sourceCode} depending on the current + component's build settings. The choice would be exported in binary + modules, but for cleanliness we would also require public APIs to + visibly commit to one choice or the other. + +- `WeakReference`{.sourceCode}: a fragile value type with a single + public `weak`{.sourceCode} field of type `T`{.sourceCode}. As above, + there should be an implicit conversion *from* `T`{.sourceCode} but + no implicit conversion to `T`{.sourceCode} (or even + `Optional`{.sourceCode}). There is, however, no need for safe and + unsafe variants. + +The library should consider providing the following types: + +- A simple, memory-sensitive weak cache. +- `Finalizer`{.sourceCode}: a value type which is constructed with a + referent and a `() -> ()`{.sourceCode} function and which causes the + function to be run (on a well-known dispatch queue?) when the object + is finalized. diff --git a/docs/weak.rst b/docs/weak.rst deleted file mode 100644 index 55079a9cc06d2..0000000000000 --- a/docs/weak.rst +++ /dev/null @@ -1,1164 +0,0 @@ -:orphan: - -================= - Weak References -================= - -:Author: John McCall -:Date: 2013-05-23 - -**Abstract:** This paper discusses the general concept of weak -references, including various designs in other languages, and proposes -several new core language features and a more sophisticated runtime -support feature which can be exploited in the standard library. - -.. warning:: This document was used in planning Swift 1.0; it has not been kept - up to date and does not describe the current or planned behavior of Swift. - -Reference Graphs -================ - -Let's run through some basic terminology. - -Program memory may be seen abstractly as a (not necessarily connected) -directed graph where the nodes are objects (treating transient -allocations like local scopes as objects) and the edges are -references. (Multi-edges and self-edges are permitted, of course.) - -We may assign a level of strength to each of these edges. We will -call the highest level *strong*; all others are some flavor of *weak*. - -Every object has a *strength of reference* which arises from the -reference graph. This can be any level of strength or the special -value *unreferenced*. The strength of a path is the minimum strength -of any edge in the path. The strength of a set of paths is the -maximum strength of all the paths, unless the set is empty, in which -case it is *unreferenced*. - -In general, the implementation is only outright forbidden to -deallocate an object if it is strongly referenced. However, -being somehow weakly referenced may trigger some sort of additional -guarantee; see the Language Precedents section. - -In a cycle-collecting environment, certain nodes are given special -treatment as *roots*; these nodes are always strongly referenced. -Otherwise, the strength of reference for an object is the strength -of all paths from any root to the object. - -In a non-cycle-collecting environment, the strength of reference for -an object is the strength of all the direct references to that -object, taken as length=1 paths. Note that this environmental -consideration becomes a language guarantee: even if the implementation -can trivially prove that an object is referenced only by itself, it -is still not permitted to deallocate the object. - -It is common for certain kinds of reference to not receive a full -guarantee. For example, a strong reference from a local variable -may lose effectiveness as soon as the variable is no longer needed -(but before it formally leaves scope). This is pervasive in GC -environments but also true in e.g. ObjC ARC. - -In some programs, especially event-driven programs like UIs or -servers, it can be useful to consider the *static* reference graph as -captured during a notional steady state. Local scopes may form many -references to objects in the static graph, but unless the scope -persists indefinitely or a major restructuring is triggered, these -references are likely to have no effect on the reference graph, and so -their strength of reference has no semantic importance. Understanding -this helps to explain why many environments provide no direct way to -form a local weak reference. - -Language and Library Precedents -=============================== - -We're only going to discuss *managed* precedents here. - -It is possible to create a kind of weak reference by just not -providing any special behavior to an object reference; if the -object is deallocated, the reference will dangle and uses of it -will likely crash or cause corruption. This could happen by -e.g. declining to insert reference-count manipulations for a -particular variable (in an ARC-like environment) or not mapping -a variable's frame offset as a pointer in a type map (in a -precise-GC environment). We will call this *dangling weak*. - -Objective-C ------------ - -All modes of Objective-C automate memory management for -synthesized properties. In GC and ARC, accessors just -use the normal semantics for the underlying ivar (plus -copy/atomic guarantees, of course). In MRC, accessors -use dangling weak semantics unless they're :code:`retain` -or :code:`copy`, in which case they maintain a +1 refcount -invariant on the referent. - -In GC and ARC, variables qualified with :code:`__weak` are -immediately zeroed out when the referenced object begins -deallocation. There is no syntactic difference on use; -it's just possible that the value read will be :code:`nil` -instead of whatever was last written there, possibly causing -the loading code to crash (e.g. on an ivar access) or silently -do nothing (e.g. on a message send). There is an opt-in -warning in ARC for certain uses of values loaded from -:code:`__weak` ivars. - -In GC, it is also possible to construct a dangling -weak reference by storing an object pointer into (1) unscanned -heap memory or (2) an instance variable that's not of -object-pointer type and isn't qualified with :code:`__strong` -or :code:`__weak`. Otherwise, object references are strong -(including all references from local scopes). - -In ARC, it is possible to construct a dangling weak reference -by using the :code:`__unsafe_unretained` qualifier or by -bridging a pointer value to a C pointer type. - -C++ ---- - -C++ smart pointers (e.g. :code:`std::unique_ptr`) typically -permit the creation of a dangling-weak reference by -providing an accessor to get the pointer as a normal C -pointer. (Even if they didn't have :code:`get()`, you could -manually call :code:`operator->` to get the same effect.) - -C++'s :code:`std::shared_ptr` permits the formation of -weak pointers (:code:`std::weak_ptr`) from shared pointers. -It is not possibly to directly use a weak pointer; it must -first be converted back to a :code:`shared_ptr`, either by -using the :code:`lock()` operation (which produces a null -pointer if the referent has been deallocated) or by directly -constructing a :code:`shared_ptr` with the :code:`weak_ptr` -(which throws an exception if the referent has been deallocated). -There is also a way to explicitly query whether a -:code:`weak_ptr` is still valid, which may be more efficient -than checking the result of the cast. - -Java ----- - -Java does not provide any facility for dangling weak references. -The standard library does provide three levels of weak reference -(in :code:`java.lang.ref`). References cannot be re-seated -(although they can be explicitly cleared), and users must call -:code:`get()` in order to access the value, which may yield -:code:`null`. - -There is a great deal of interesting discussion of these -reference classes `here `_. - -Java :code:`Reference` objects may be constructed with an -optional :code:`ReferenceQueue`; if so, then when the -object's reachability changes, the reference object will be -added to that queue. This permits data structures to clean -up after cleared soft references without needing to either -periodically scan the entire structure or be fully lazy. -Additional data may be added to the reference object by -subclassing it. - -The references are presented in order of decreasing strength. - -:code:`SoftReference` is a sort of quasi-strong reference -which holds onto the object until the VM begins to run out -of memory. Soft references to softly-referenced objects are -guaranteed to have been cleared before the VM can throw an -:code:`OutOfMemoryError`. The reference will be cleared -before it is added to its reference queue (and so the -reference queue cannot resurrect the object). The intent -of soft references is to enable memory-sensitive caches, -but in practice a memory-sensitive cache would probably -want to implement a more subtle replacement strategy than -"drop things at random as soon as memory runs low". A -more interesting use is a memory-guided circuit-breaker: -when building up a very large structure, hold it in a -soft reference, and if that references goes null during -construction, just bail out. But that's a pretty tricky -use-case to get right. - -:code:`WeakReference` is intended for use in non-memory-sensitive -weak caches, like a uniquing cache; it persists only as long -as the referent is more strongly referenced. The reference -will be cleared before it is added to its reference queue (and -so the reference queue cannot resurrect the object). - -:code:`PhantomReference` provides a way to attach extra -finalization to an object without actually using finalizers -(which have several problems, including the ability to -resurrect the object). The phantom reference *always* -presents :code:`null` as its value and is therefore itself -useless as a reference. Phantom references are enqueued -after the object is finalized and therefore at a point when -there can be no references to the object within the VM -at all. However, the object itself cannot be deallocated -until the phantom references are all cleared or themselves -deallocated, which I believe is for the convenience of native -code that may hold a dangling weak reference to the referent -(or which may be able to directly read the reference). - -.NET ----- - -The :code:`WeakReference` class in .NET is similar to -Java's :code:`WeakReference` class in that the value -cannot be accessed directly; it must be accessed -via the :code:`Target` property, which may yield -:code:`null`. The reference may be reseated to a -different value. - -Weak references may be created *long*, which permits the -target object to be finalized but not actually deallocated. - -Python ------- - -A :code:`weakref` acts like a function object; it is created -with a particular value, which cannot be reseated. The -function will yield :code:`None` if the referent is collected. - -There is library functionality to automatically proxy a value -as a weak reference. An exception is thrown if an operation -is performed on the proxy but the referent has been collected. - -A :code:`weakref` may be constructed with a callback function. -The callback will be called after the weak reference is cleared; -it is, however, passed the weak ref object itself. - -Ruby ----- - -A :code:`WeakRef` is automatically a proxy for an object. -There is a :code:`weakref_alive` method to query whether the -reference is still alive; another other operation will cause -an exception to be thrown. - -Rust ----- - -As far as I can tell, there is nothing like a weak reference -in Rust at the moment. - -A *managed pointer* (:code:`@int`) is a strong reference -subject to GC. - -An *owning pointer* (:code:`~int`) is a strong reference -that cannot be cloned (copying the pointer actually copies the -underlying data). - -A *borrowed pointer* (:code:`&int`) is essentially a dangling -weak reference that is subject to static restrictions which -ensure that it doesn't actually dangle. It is thus primarily -a performance optimization. - -A *raw pointer* (:code:`*int`) is a dangling weak reference. - -Haskell -------- - -Yes, of course Haskell has weak references. - -A :code:`Weak t` is an association between a hidden key -and a visible value of type :code:`t`. -:code:`doRefWeak theRef` is an :code:`IO (Maybe t)`. - -A weak reference may be constructed with an optional -:code:`IO ()` which will be run when the referent is -collected. This finalizer may (somehow) refer to the key -and value without itself keeping them alive; it is also -explicitly permitted to resurrect them. - - -Use Cases -========= - -There are many problems that are potentially addressable with -functionality like weak references. It is not at all obvious -that they should be addressed with the same language feature. - -Back references ---------------- - -Given that Swift is not cycle-collecting, far and away the most -important use case in the static reference graph is that of the -*back-reference*: a reference *R* to an object which holds a strong -reference (possibly indirectly) to the object holding *R*. Examples -include: - -- A 'previousNode' pointer in a doubly-linked list. - -- A 'parent' pointer in a render tree. - -- An edge in a general graph structure. - -These have several properties in common: - -- Using strong references would require a lot of explicit - code to tear down the reference cycles. - -- These references may be accessed very frequently, so - performance is important. - -- It is not always feasible to make these references valid - immediately on construction. - -- Traversing a reference after the referent is deallocated is likely a - sign that something has been kept alive longer than it was meant to - be. However, programmers may reasonably differ about the correct - response to this: crashing, and therefore encouraging the programmer - to track down a root cause, or simply writing the operation to - handle both cases correctly. Ultimately, this choice comes down to - philosophy. - -Caches ------- - -Weak caches are used in order to prevent a cache from taking -over all available memory. By being tied to the reachability -of a value, the cache prevents entries from spuriously -expiring when their values are still in active use; but by -using weak references, the cache permits the system to -deallocate values that are no longer in use. - -Generally, a data structure using weak references extensively -also needs some way to receive notification that the weak -reference was collected. This is because entries in the data -structure are likely to have significant overhead even if the -value is collected. A weak data structure which receives no -notification that a reference has been invalidated must either -allow these entries to accumulate indefinitely or must -periodically scan the entire structure looking for stale entries. - -A weak reference which permits immediate deallocation of its -referent when the last strong reference is dropped is -substantially less useful for the implementation of a weak -cache. It is a common access pattern (for, say, a memoizing -cache) for a value to be looked up many times in rapid -succession, but for each use to be temporarily disjoint -from the others. A naive use of weak references in this case -will simply cause the cache to thrash. This problem is less -likely to arise in an environment with nondeterministic -collection because the entry is likely to service multiple -lookups between collections. - -It is likely that users implementing weak data structures -would prefer a highly flexible infrastructure centered around -resurrection and notifications of reaching a zero refcount -than a more rigid system built directly into the language. -Since the Swift model is built around statically-inserted -operations rather than a memory scanner, this is much more -workable. - -External Finalization ---------------------- - -Finalization models built around calling a method on the -finalized object (such as Objective-C's :code:`-dealloc`) -suffer from a number of limitations and problems: - - - Since the method receives a pointer to the object being - deallocated, the implementation must guard against - attempts to resurrect the object. This may complicate - and/or slow down the system's basic reference-management - logic, which tends to be quite important for performance. - - - Since the method receives a pointer to the object being - deallocated, the implementation must leave the object at - least a minimally valid state until the user code is - complete. For example, the instance variables of a - subclass cannot be destroyed until a later phase of - destruction, because a superclass finalizer might invoke - subclass behavior. (This assumes that the dynamic type - of the object does not change during destruction, which - is an alternative that brings its own problems.) - - - Finalization code must be inherent to the object; other - objects cannot request that code be run when the object - is deallocated. For example, an object that registers - itself to observe a certain event source must explicitly - deregister itself in a finalizer; the event source cannot - simply automatically drop the object when it is - deallocated. - -Optimization ------------- - -Functions often create a large number of temporary references. In a -reference-counting environment like Swift, these references require -the implementation to implicitly perform operations to increment and -decrement the reference count. These operations can be quite fast, -but they are not free, and our experience has been that the -accumulated cost can be quite significant. A straightforward local -static analysis can eliminate many operations, but many others will be -blocked by abstraction barriers, chiefly dynamically-dispatched calls. -Therefore, if Swift is to allow precise performance control, it is -important to be able to allow motivated users to selectively control -the emission of reference-counting operations. - -This sort of control necessarily permits the creation of dangling weak -references and so is not safe. - -Proposal Overview -================= - -Looking at these use-cases, there are two main thrusts: - - - There is a general need to set up back references to objects. - These references must be designed for convenient use by non-expert - users. - - - There are a number of more sophisticated use cases which require - notification or interruption of deallocation; these can be used in - the implementation of higher-level abstractions like weak caches. - Here it is reasonable to expect more user expertise, such that - power and flexibility should take priority over ease of use. - -The second set of use cases should addressed by library types working -on top of basic runtime support. - -The first set of use cases will require more direct language support. -To that end, I propose adding two new variable attributes, -:code:`@weak` and :code:`@unowned`. I also propose a small slate of -new features which directly address the problem of capturing a value -in a closure with a different strength of reference than it had in the -enclosing context. - -Proposed Variable Attributes -============================ - -In the following discussion, a "variable-like" declaration is any -declaration which binds a name to a (possibly mutable) value of -arbitrary type. Currently this is just :code:`var`, but this proposal -also adds :code:`capture`, and we may later add more variants, such as -:code:`const` or :code:`val` or the like. - -:code:`@weak` --------------- - -:code:`weak` is an attribute which may be applied to any -variable-like declaration of reference type :code:`T`. For -type-system purposes, the variables behaves like a normal -variable of type :code:`Optional`, except: - - - it does not maintain a +1 reference count invariant and - - - loading from the variable after the current referent (if present) - has started destruction will result in a :code:`Nothing` value, - indistinguishable from the normal case. - -The semantics are quite similar to weak references in other -environments (particularly Objective-C) except that the change in -formal type forces the user of the value to check its validity before -using it. - -It doesn't really matter how the compiler would find the type -:code:`Optional`; compiler-plus-stdlib magic, most likely. I do -not think the type should be :code:`Weak` because that would -effectively make this a type attribute and thus make it too easy to -accidentally preserve the value as a weak reference. See the section -discussing type attributes vs. declaration attributes. - -Giving the variable a consistent type of :code:`Optional` permits -the user to assign :code:`Nothing` into it and therefore removes the -somewhat odd expressive possibility of a reference that can only be -missing if the object has been deallocated. I think this is an -acceptable cost of making a cleaner feature. - -One alternative to using :code:`Optional` would be to simply treat -the load as a potentially-failing operation, subject to the (not yet -precisely designed) language rules for error handling. This would -require the runtime to potentially synthesize an error event, which -could then propagate all the way to the end-user if the programmer -accidentally failed to check the result in a context that permitted -error propagation outwards. That's bad. - -A slightly different alternative would be to treat it as a form of -error orthogonal to the standard user-error propagation. This would -be cleaner than changing the type of the variable but can't really be -designed without first having a solid error-handling design. - - -:code:`@unowned` ------------------ - -:code:`unowned` is an attribute which may be applied to any -"variable-like" declaration of reference type :code:`T`. For -type-system purposes, the variable behaves exactly like a normal -variable of type :code:`T`, except: - - - it does not maintain a +1 reference count invariant and - - - loading from the variable after the referent has started - destruction causes an assertion failure. - -This is a refinement of :code:`weak` focused more narrowly on the case -of a back reference with relatively tight validity invariants. This -is also the case that's potentially optimizable to use dangling weak -references; see below. - -This name isn't really optimal. We've considered several different -candidates: - - - :code:`weak` is a poor choice because our semantics are very - different from weak references in other environments where it's - valid to access a cleared reference. Plus, we need to expose - those semantics, so the name is claimed. - - - :code:`backref` is strongly evocative of the major use case in the - static reference graph; this would encourage users to use it for - back references and to consider alternatives for other cases, both - of which I like. The latter also makes the husk-leaking - implementation (see below) more palatable. It also contrasts very - well with :code:`weak`. However, its evocativeness makes it - unwieldy to use for local reference-counting optimizations. - - - :code:`dangling` is more general than :code:`backref`, but it has - such strong negative associations that it wouldn't be unreasonable - for users to assume that it's unsafe (with all the pursuant - debugging difficulties) based on the name alone. I don't think - we want to discourage a feature that can help users build tighter - invariants on their classes. - - - :code:`unowned` is somewhat cleaner-looking, and it isn't as tied - to a specific use case, but it does not contrast with :code:`weak` - *at all*; only someone with considerable exposure to weak - references would understand why we named each one the way we did, - and even they are likely to roll their eyes at us. But it's okay - for a working proposal. - -Asserting and Uncheckable -......................... - -There should not be a way to check whether a :code:`unowned` -reference is still valid. - -- An invalid back-reference is a consistency error that - we should encourage programmers to fix rather than work - around by spot-testing for validity. - -- Contrariwise, a weak reference that might reasonably be - invalidated during active use should be checked for validity - at *every* use. We can provide a simple library facility - for this pattern. - -- Permitting implicit operations like loads to fail in a - recoverable way may end up complicating the language model - for error-handling. - -- By disallowing recovery, we create a model where the only - need to actually register the weak reference with the system - is to enable a consistency check. Users who are confident - in the correctness of their program may therefore simply - disable the consistency check without affecting the semantics - of the program. In this case, that leaves the variable a - simple dangling-weak reference. - -Implementation -.............. - -The standard implementation for a weak reference requires the -address of the reference to be registered with the system so -that it can be cleared when the referent is finalized. This -has two problems: - -- It forces the runtime to maintain a side-table mapping - objects to the list of weak references; generally this - adds an allocation per weakly-referenced object. - -- It forces the representation of weak references to either - be non-address-invariant or to introduce an extra level of - indirection. - -For some use cases, this may be warranted; for example, in -a weak cache it might come out in the noise. But for a simple -back-reference, these are substantial penalties. - -Dave Z. has proposed instead using a weak refcount, analogous to a -strong refcount. Ownership of a weak retain can be easily transferred -between locations, and it does not require a side-table of an object's -weak references. However, it does have a very important downside: -since the system cannot clear all the references, it is impossible to -actually deallocate an object that is still weakly-referenced -(although it can be finalized). Instead, the system must wait for all -the weak references to at least be accessed. We call this "husk -leaking". - -This downside could be a problem for a general weak reference. -However, it's fine for a back-reference, which we expect to typically -be short-lived after its referent is finalized. - -Declaration Attribute or Type Attribute ---------------------------------------- - -This proposal describes :code:`weak` and :code:`unowned` as -declaration attributes, not type attributes. - -As declaration attributes, :code:`@unowned` and :code:`weak` would be -permitted on any :code:`var` declaration of reference type. Their -special semantics would be a property only of the declaration; in -particular, they would not change the type (beyond the shift to -:code:`Optional` for :code:`weak`) , and more generally the -type-checker would not need to know about them. The implementation -would simply use different behavior when loading or storing that -variable. - -As a type attribute, :code:`weak` and :code:`@unowned` would be -permitted to appear at arbitrary nested locations in the type system, -such as tuple elements, function result types, or generic arguments. -It would be possible to have both lvalues and rvalues of so-qualified -type, and the type checker would need to introduce implicit -conversions in the right places. - -These implicit conversions require some thought. Consider code like -the following:: - - func moveToWindow(newWindow : Window) { - var oldWindow = self.window // an @unowned back reference - oldWindow.hide() // might remove the UI's strong reference - oldWindow.remove(self) - newWindow.add(self) - } - -It would be very unfortunate if the back-reference nature of the -:code:`window` property were somehow inherited by :code:`oldWindow`! -Something, be it a general rule on loading back-references or a -type-inference rule, must introduce an implicit conversion and cause -the :code:`unowned` qualifier to be stripped. - -That rule, however, is problematic for generics. A key goal of -generics is substitutability: the semantics of generic code should -match the semantics of the code you'd get from copy-pasting the -generic code and substituting the arguments wherever they're written. -But if a generic argument can be :code:`@unowned T`, then this -won't be true; consider:: - - func foo(x : T) { - var y = x - ... - } - -In substituted code, :code:`y` would have the qualifier stripped and -become a strong reference. But the generic type-checker cannot -statically recognize that this type is :code:`unowned`-qualified, so -in order to match semantics, this decision must be deferred until -runtime, and the type-checker must track the unqualified variant of -:code:`T`. This adds a great deal of complexity, both to the -implementation and to the user model, and removes many static -optimization opportunities due to potential mismatches of types. - -An alternative rule would be to apply an implicit "decay" to a strong -reference only when a type is known to be a :code:`unowned` type. It -could be argued that breaking substitution is not a big deal because -other language features, like overloading, can already break it. But -an overlapping overload set with divergent behavior is a poor design -which itself violates substitution, whereas this would be a major -unexcused deviation. Furthermore, preserving the weakness of a -reference is not a safe default, because it permits the object to be -destroyed while a reference is still outstanding. - -In addition, any implementation model which permits the safety checks -on :code:`unowned`\ s to be disabled will require all code to agree about -whether or not the checks are enabled. When this information is tied -to a declaration, this is easy, because declarations are ultimately -owned by a particular component, and we can ask how that component is -compiled. (And we can demand that external APIs commit to one level -of safety or the other before publishing.) The setting for a type, on -the other hand, would have to be determined by the module which "wrote -the type", but again this introduces a great deal of complexity which -one can only imagine settling on the head of some very confused user. - -For all these reasons, I feel that making :code:`weak` and -:code:`unowned` type attributes would be unworkable. However, there -are still costs to making them declaration attributes: - -- It forces users to use awkward workarounds if they want to - make, say, arrays of back-references. - -- It makes back-references less composable by, say, preventing - them from being stored in a tuple. - -- It complicates SIL and IR-gen by making the rules for manipulating a - physical variable depend on more than just the type of the variable. - -- It automatically enables certain things (like passing the address of - a :code:`@unowned` variable of type :code:`T` to a :code:`inout T` - parameter) that perhaps ought to be more carefully considered. - -The first two points can be partly compensated for by adding library -types to wrap a back-reference. Accessing a wrapped reference will -require extra syntax, and it runs the same risk of accidentally -preserving the weakness of a reference that I discussed above. -However, note that these problems are actually at odds: requiring -extra syntax to access the wrapped reference will leave breadcrumbs -making it clear when the change-over occurs. For more on this, -see the library section. - -:code:`weak`-Capable Types --------------------------- - -Swift reference types can naturally be made to support any kind of -semantics, and I'm taking it on faith that we could enhance ObjC -objects to support whatever extended semantics we want. There -are, however, certain Swift value types which have reference-like -semantics that it could be useful to extend :code:`weak` (and/or -:code:`unowned`) to: - -- Being able to conveniently form an optional back-reference seems - like a core requirement. If :code:`weak` were a type attribute, - we could just expect users to write :code:`Optional<@weak T>`; - as a declaration attribute, this is substantially more difficult. I - expect this to be common enough that it'll be unreasonable to ask - users to use :code:`Optional>`. - -- Being able to form a back-reference to a slice or a string seems - substantially less important. - -One complication with extending :code:`weak` to value types is that -generally the implementing type will need to be different from the -underlying value type. Probably the best solution would be to hide -the use of the implementing type from the type system outside of the -well-formedness checks for the variable; SIL-gen would lower the field -to its implementing type using the appropriate protocol conformances. - -As long as we have convenient optional back-references, though, we -can avoid designing a general feature for 1.0. - - -Generic Weak Support --------------------- - -All other uses for weak references can be glossed as desiring -some amount of additional work to occur when the strong reference -count for an object reaches zero. This necessarily entails a -global side-table of such operations, but I believe that's -acceptable as long as it's relegated to less common use-cases. - -It is important that the notification mechanism not require -executing code re-entrantly during the finalization process. - -I suggest adopting an interface centered around the Java -concept of a :code:`ReferenceQueue`. A reference structure -is registered with the runtime for a particular object with -a particular set of flags and an optional reference queue:: - - struct Reference { - void *Referent; // must be non-null upon registration - struct ReferenceQueue *Queue; // must be valid or null - size_t Reserved[2]; - }; - - void swift_registerReference(struct Reference *reference, - size_t flags); - -The user/library code is responsible for allocating these structures -and initializing the first two fields, and it may include arbitrary -fields before or after the :code:`Reference` section, but while the -reference is registered with the runtime, the entire :code:`Reference` -section becomes reserved and user/library code must not access it in -any way. - -The flags include: - -- A priority. Should be constrained to two or three bits. References - are processed in order of decreasing priority; as long as a - reference still exists with higher priority, references with lower - priority cannot be processed. Furthermore, as long as any reference - exists, the referent cannot be finalized. - -- Whether to automatically clear the reference when processing it. - Note that a cleared reference is still considered to be - registered with the runtime. - -These could be combined so that e.g. even priorities cause -an automatic clear and odd priorities do not; this would avoid some -odd effects. - -The runtime may assume that explicit user operations on the same -reference will not race with each other. However, user operations on -different references to the same referent may be concurrent, either -with each other or with other refcount operations on the referent. - -The operations on references are as follows:: - - void *swift_readReference(struct Reference *reference); - -This operation atomically either produces a strong reference to the -referent of the given object or yields :code:`null` if the referent -has been finalized (or if the referent is :code:`null`). The -reference must currently be registered with the runtime:: - - void swift_writeReference(struct Reference *reference, - void *newReferent); - -This operation changes the referent of the reference to a new object, -potentially :code:`null`. The argument is taken at +0. The reference -must currently be registered with the runtime. The reference keeps -the same flags and reference queue:: - - void swift_unregisterReference(struct Reference *Reference); - -This operation clears a reference, removes it from its reference -queue (if it is enqueued), and unregisters it from the runtime. -The reference must currently be registered with the runtime. - -I propose the following simple interface to a ReferenceQueue; -arguably, however, it ought to be a reference-counted library -type with a small amount of native implementation:: - - struct ReferenceQueue; - struct ReferenceQueue *swift_createReferenceQueue(void); - -Allocate a new reference queue:: - - void swift_destroyReferenceQueue(struct ReferenceQueue *queue); - -Destroy a reference queue. There must not be any references with -this queue currently registered with the runtime:: - - struct Reference *swift_pollReferenceQueue(struct ReferenceQueue *queue); - - -Proposed Rules for Captures within Closures -=========================================== - -When a variable from an enclosing context is referenced in a closure, -by default it is captured by reference. Semantically, this means that -the variable and the enclosing context(s) will see the variable as a -single, independent entity; modifications will be seen in all places. -In terms of the reference graph, each context holds a strong reference -to the variable itself. (In practice, most local variables captured -by reference will not require individual allocation; usually they will -be allocated as part of the closure object. But in the formalism, -they are independent objects.) - -Closures therefore make it fairly easy to introduce reference cycles: -for example, an instance method might install a closure as an event -listener on a child object, and if that closure refers to -:code:`self`, a reference cycle will be formed. This is an -indisputable drawback of an environment which cannot collect reference -cycles. - -Relatively few languages both support closures and use -reference-counting. I'm not aware of any that attempt a language -solution to the problem; the usual solution is to the change the -captured variable to use weak-reference semantics, usually by copying -the original into a new variable used only for this purpose. This is -annoyingly verbose, clutters up the enclosing scope, and duplicates -information across multiple variables. It's also error-prone: since -both names are in scope, it's easy to accidentally refer to the wrong -one, and explicit capture lists only help if you're willing to be very -explicit. - -A better alternative (which we should implement in Objective-C as -well) is to permit closures to explicitly specify the semantics under -which a variable is captured. In small closures, it makes sense to -put this near the variable reference; in larger closures, this can -become laborious and redundant, and a different mechanism is called for. - -In the following discussion, a *var-or-member expression* is an -expression which is semantically constrained to be one of: - - - A reference to a local variable-like declaration from an - enclosing context. - - - A member access thereof, possibly recursively. - -Such expressions have two useful traits: - - - They always end in an identifier which on some level meaningfully - identifies the object. - - - Evaluating them is relatively likely (but not guaranteed) to not - have interesting side effects, and so we feel less bad about - apparently shifting their evaluation around. - -Decorated Capture References ----------------------------- - -Small closures are just as likely to participate in a reference cycle, -but they suffer much more from extraneous syntax because they're more -likely to be center-embedded in interesting expressions. So if -there's anything to optimize for in the grammar, it's this. - -I propose this fairly obvious syntax:: - - button1.setAction { unowned(self).tapOut() } - button2.setAction { if (weak(self)) { weak(self).swapIn() } } - -The operand is evaluated at the time of closure formation. Since -these references can be a long way from the top of the closure, we -don't want to allow a truly arbitrary expression here, because the -order of side-effects in the surrounding context could be very -confusing. So we require the operand to be an :code:`expr-var-or-member`. -More complicated expressions really ought to be hoisted out to a -separate variable for legibility anyway. - -I do believe that being able to capture the value of a property -(particularly of :code:`self`) is very important. In fact, it's -important independent of weak references. It is often possible to -avoid a reference cycle by simply capturing a specific property value -instead of the base object. Capturing by value is also an -expressivity improvement: the programmer can easily choose between -working with the property's instantaneous value (at the time the -closure is created) or its current value (at the time the closure is -invoked). - -Therefore I also suggest a closely-related form of decoration:: - - button3.setAction { capture(self.model).addProfitStep() } - -This syntax would force :code:`capture` to become a real keyword. - -All of these kinds of decorated references are equivalent to adding a -so-attributed :code:`capture` declaration (see below) with a nonce -identifier to the top of the closure:: - - button1.setAction { - capture @unowned _V1 = self - _V1.tapOut() - } - button2.setAction { - capture @weak _V2 = self - if (_V2) { _V2.swapIn() } - } - button3.setAction { - capture _V3 = self.model - _V3.addProfitStep() - } - -If the operand of a decorated capture is a local variable, then that -variable must not be the subject of an explicit :code:`capture` -declaration, and all references to that variable within the closure -must be identically decorated. - -The requirement to decorate all references can add redundancy, but -only if the programmer insists on decorating references instead of -adding an explicit :code:`capture` declaration. Meanwhile, that -redundancy aids both maintainers (by preventing refactors from -accidentally removing the controlling decoration) and readers (who -would otherwise need to search the entire closure to understand how -the variable is captured). - -Captures with identical forms (the same sequence of members of the -same variable) are merged to the same capture declaration. This -permits type refinement to work as expected, as seen above with the -:code:`weak` capture. It also guarantees the elimination of some -redundant computation, such as with the :code:`state` property in this -example:: - - resetButton.setAction { - log("resetting state to " + capture(self.state)) - capture(self.model).setState(capture(self.state)) - } - -I don't see any immediate need for other kinds of capture decoration. -In particular, I think back references are likely to be the right kind -of weak reference here for basically every use, and I don't think that -making it easy to capture a value with, say, a zeroable weak reference -is important. This is just an intuition deepened by hallway -discussions and close examination of a great many test cases, so I -concede it may prove to be misguided, in which case it should be easy -enough to add a new kind of decoration (if we're willing to burn a -keyword on it). - -:code:`capture` Declarations ----------------------------- - -This feature generalizes the above, removing some redundancy in large -closures and adding a small amount of expressive power. - -A :code:`capture` declaration can only appear in a closure: an -anonymous closure expression or :code:`func` declaration that appears -directly within a function. (By "directly" I mean not within, say, a -local type declaration within the function). :code:`capture` will -need to at least become a context-sensitive keyword. - -A closure may contain multiple :code:`capture` declarations, but they -must all appear at the very top. One reason is that they can affect -the capture semantics within the closure, so someone reading the -closure should be able to find them easily. Another reason is that -they can involve executing interesting code in the enclosing context -and so should reliably appear near the closure formation site in the -source code:: - - decl ::= decl-capture - decl-capture ::= 'capture' attribute-list '=' expr-var-or-member - decl-capture ::= 'capture' attribute-list decl-capture-expr-list - decl-capture-expr-list ::= expr-var-or-member - decl-capture-expr-list ::= expr-var-or-member ',' decl-capture-expr-list - -Both forms of :code:`capture` declaration cause one or more fields to -be created within the closure object. At the time of creating the -closure, these fields are initialized with the result of evaluating an -expression in the enclosing context. Since the expression is -evaluated in the enclosing context, it cannot refer to "previous" -captures; otherwise we could have some awkward ambiguities. I think -we should reserve the right to not execute an initializer if the -closure will never be called; this is more important for local -:code:`func` declarations than for anonymous closure expressions. - -The fields introduced by :code:`capture` declarations should be -immutable by default, but programmers should be able to write -something like :code:`capture var ...` to make them mutable. Locking -down on mutation isn't quite as important as it is with implicit -captures (where it's easy to accidentally believe you're modifying the -enclosing variable) or even explicit captures in C++11 lambdas (where -copies of the lambda object will copy the capture field and thus -produce mystifying behavior in uncareful code), but it's still a -source of easy mistakes that should require manual intervention to -enable. This all presumes that we eventually design mutability into -the language, of course. - -In the pattern-initializer form, the field names are given explicitly -by the pattern. The abbreviated form borrows the name of the captured -member or local variable. In either case, names should be subject to -the usual shadowing rules, whatever they may be, with the exception -that capturing an enclosing variable with the abbreviated form is not -problematic. - -Attributes on a :code:`capture` declaration affect all the fields it -declares. - -Let's spell out some examples. I expect the dominant form to be a -simple identifier:: - - capture @unowned foo - -This captures the current value of whatever :code:`foo` resolves to -(potentially a member of :code:`self`!) and binds it within the -closure as a back-reference. - -Permitting the slightly more general form:: - - capture window.title - -allows users to conveniently capture specific values without mucking -up the enclosing scope with tons of variables only needed for setting -up the closure. In particular, this makes it easy to capture specific -fields out of an enclosing :code:`self` object instead of capturing -the object itself; that, plus forcing uses of :code:`self` to be -explicit in closures, would help users to conveniently avoid a class -of inadvertent retain cycles. - -I've included the general pattern-initializer form mostly for ease of -exposition. It adds no major expressivity improvements over just -creating a variable in the enclosing context. It does avoid -cluttering the enclosing scope with new variables and permits captures -to be locally renamed, and it can very convenient if introducing a new -variable in the enclosing scope would be complicated (for example, if -there were a reason to prefer using a single statement there). I -don't think it does any harm, but I wouldn't mourn it, either. I do -think that generalizing the initializer to an arbitrary expression -would be a serious mistake, because readers are naturally going to -overlook code which occurs inside the closure, and promoting side -effects there would be awful. - -It would be nice to have a way to declare that a closure should not -have any implicit captures. I don't have a proposal for that right now, -but it's not important for weak references. - -Nested Closures ---------------- - -It is important to spell out how these rules affect captures in nested -closures. - -Recall that all of the above rules can be transformed into -``capture`` declarations at the beginning of a closure, and that -``capture`` declarations always introduce a by-value capture -instead of a by-reference capture. - -A by-reference capture is always of either a local variable or a -``capture`` declaration. In neither case do we currently permit -such captures to "drag in" other declarations silently, to the extent -that this is detectable. This means that mutable ``capture`` -declarations that are themselves captured by reference must be -separately allocated from the closure object, much like what happens -with normal locals captured by reference. - -I've considered it quite a bit, and I think that a by-value capture of -a variable from a non-immediately enclosing context must be made -ill-formed. At the very least, it must be ill-formed if either the -original variable is mutable (or anything along the path is, if it -involves properties) or the capture adds ``@unowned``. - -This rule will effectively force programmers to use extra variables or -``capture``\ s as a way of promising validity of the internal -captures. - -The motivation for this prohibition is that the intent of such -captures is actually quite ambiguous and/or dangerous. Consider -the following code:: - - async { doSomething(); GUI.sync { unowned(view).fireCompleted() } } - -The intent of this code is to have captured a back-reference to the -value of :code:`view`, but it could be to do so at either of two -points in time. The language must choose, and in this hypothetical it -must do so without further declaration of intent and without knowledge -of when and how many times the closures will be called. - -Suppose that we capture the value at the earlier point, when we form -the outer closure. This will behave very surprisingly if :code:`view` -is in fact mutated; it may be easier to imagine this if, instead of a -simple local variable, it was instead a path like :code:`self.view`. -And it's not clear that forming a back-reference at this earlier point -is actually valid; it is easy to imagine code that would rely on the -intermediate closure holding a strong reference to the view. -Furthermore, and crucially, these issues are inextricable: we cannot -somehow keep track of the mutable variable but only hold its value -weakly. - -But suppose instead that we capture the value at the later point. -Then the intermediate closure will capture the :code:`view` variable -by reference, which means that in effect it will hold :code:`view` -strongly. This may actually completely subvert the user's desired -behavior. - -It's not clear to me right now whether this problem applies equally to -explicit :code:`capture` declarations. Somehow decorated expressions -seem more ambiguous in intent, probably because the syntax is more -thoughtlessly convenient. On the other hand, making the decoration -syntax not just a shorthand for the explicit declarations makes the -model more complex, and it may be over-complex already. - -So in summary, it would be best to enforce a strong prohibition against -these dangerous multi-level captures. We can tell users to introduce -secondary variables when they need to do subtle things across several -closure levels. - - -Library Directions -================== - -The library should definitely provide the following types: - -- :code:`UnownedReference`: a fragile value type with a single - public :code:`unowned` field of type :code:`T`. There should be an - implicit conversion *from* :code:`T` so that obvious initializations - and assignments succeed. However, there should not be an implicit - conversion *to* :code:`T`; this would be dangerous because it could - hide bugs introduced by the way that e.g. naive copies into locals - will preserve the weakness of the reference. - - In keeping with our design for :code:`unowned`, I think this type - should should actually be an alias to either - :code:`SafeUnownedReference` or :code:`UnsafeUnownedReference` - depending on the current component's build settings. The choice - would be exported in binary modules, but for cleanliness we would - also require public APIs to visibly commit to one choice or the - other. - -- :code:`WeakReference`: a fragile value type with a single public - :code:`weak` field of type :code:`T`. As above, there should be an - implicit conversion *from* :code:`T` but no implicit conversion to - :code:`T` (or even :code:`Optional`). There is, however, no - need for safe and unsafe variants. - -The library should consider providing the following types: - -- A simple, memory-sensitive weak cache. - -- :code:`Finalizer`: a value type which is constructed with a referent - and a :code:`() -> ()` function and which causes the function to be - run (on a well-known dispatch queue?) when the object is finalized. - diff --git a/utils/pass-pipeline/README.md b/utils/pass-pipeline/README.md new file mode 100644 index 0000000000000..5694647057015 --- /dev/null +++ b/utils/pass-pipeline/README.md @@ -0,0 +1,51 @@ +Pass Pipeline +============= + +Purpose +------- + +`Pass Pipeline` is a python library for generating pipeline json +descriptors for Swift. The goal is to enable a user to generate various +pass pipelines without needing to recompile the compiler. The main +targets for this work are: + +1. Correctness issues exposed by adding and removing parts of + pass pipelines. +2. Exploration of various pipelines from a performance perspective. + +Why not just use sil-opt and generate IR from sil for this purpose? +------------------------------------------------------------------- + +This is necessary since currently we do not have the infrastructure to +always be able to use sil-opt and friends to generate IR from a .sil +file. This is a shortcut around such issues. + +The JSON Descriptor Format +-------------------------- + +The general format is as follows: + +``` +[ + [ + "PASS_MANAGER_ID", + "run_n_times"|"run_to_fixed_point", + count, + "PASS1", "PASS2", ... + ], + ... +] +``` + +Where "id" is printed out when we process the action, "action" can be +one of "run\_n\_times", "run\_to\_fixed\_point" and "passes" is a list +of passes to run. The names to use are the stringified versions of pass +kinds. + +Structure +--------- + +There are two parts to this library. In `src`, we have the library +building blocks. This is where we define the passes and the general +pipeline infrastructure. In `scripts`, is where we implement various +pipeline generators using code from `src`. diff --git a/utils/pass-pipeline/README.rst b/utils/pass-pipeline/README.rst deleted file mode 100644 index 3fb65220056d9..0000000000000 --- a/utils/pass-pipeline/README.rst +++ /dev/null @@ -1,46 +0,0 @@ -Pass Pipeline -------------- - -Purpose -======= - -``Pass Pipeline`` is a python library for generating pipeline json descriptors -for Swift. The goal is to enable a user to generate various pass pipelines -without needing to recompile the compiler. The main targets for this work are: - -1. Correctness issues exposed by adding and removing parts of pass pipelines. -2. Exploration of various pipelines from a performance perspective. - -Why not just use sil-opt and generate IR from sil for this purpose? -=================================================================== - -This is necessary since currently we do not have the infrastructure to always be -able to use sil-opt and friends to generate IR from a .sil file. This is a -shortcut around such issues. - -The JSON Descriptor Format -========================== - -The general format is as follows: - - [ - [ - "PASS_MANAGER_ID", - "run_n_times"|"run_to_fixed_point", - count, - "PASS1", "PASS2", ... - ], - ... - ] - -Where "id" is printed out when we process the action, "action" can be one of -"run_n_times", "run_to_fixed_point" and "passes" is a list of passes to -run. The names to use are the stringified versions of pass kinds. - -Structure -========= - -There are two parts to this library. In ``src``, we have the library building -blocks. This is where we define the passes and the general pipeline -infrastructure. In ``scripts``, is where we implement various pipeline -generators using code from ``src``.