Skip to content

Commit 785481c

Browse files
robin-awsrobin-aws
authored
chore: Adopt SmithyDafnyMakefile.mk, progress towards fixing nightly build (#797)
Replaces nearly all of the `SharedMakefile.mk` with the common `smithy-dafny/SmithyDafnyMakefile.mk` makefile, just retaining configuration variables specific to this repo (such as the path to the `smithy-dafny` submodule). Uses the new features in that makefile and `smithy-dafny` itself to make the projects forwards-compatible with the latest Dafny nightly prerelease, and hence will MOSTLY fix the nightly build once merged. "Mostly" because I still need to fix some externs to make them use the pattern that avoids the Java TypeDescriptor differences between Dafny versions, but that can be fixed in a follow-up PR. Highlights of the changes: * Apply the same workflow changes as aws/aws-cryptographic-material-providers-library#195 to use `smithy-dafny` to regenerate code, either to check that the output matches what's checked in (in a new separate codegen workflow) or to be compatible with newer versions of Dafny in the nightly build (in existing workflows). * In this case we also have to locally update the MPL submodule to the latest, so that we can pick up the forwards-compatible changes to that repo, and regenerate code transitively. Generating code is unfortunately getting quite expensive, especially since the shared makefile logic isn't sophisticated enough to avoid regenerating the same code multiple times. * Because the code in this repo wasn't formatted already, but applying newer `smithy-dafny` code generation automatically formats, all the generated has trivial layout changes. * Also extracted a manual patch. * ~Applied a lot of [explicit client downcasting](https://github.com/aws/aws-database-encryption-sdk-dynamodb/pull/797/files#diff-692e2b06d124c9775e2fcd9cd9dbd10e0c8ea470e08174ed0b258b0301622581R182) to account for the change in `smithy-dafny` ~ (the change in smithy-dafny was undone so this isn't necessary any more) * Converted `Beacon.CheckBytesToHex()` from a "test lemma" to a dynamic test, because on the latest Dafny the verification got even more expensive, and this didn't seem to introduce any real additional risk.
1 parent 04c6cc4 commit 785481c

File tree

181 files changed

+7273
-2571
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

181 files changed

+7273
-2571
lines changed

‎.github/actions/polymorph_codegen/action.yml

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
#
2+
# This local action serves two purposes:
3+
#
4+
# 1. For core workflows like pull.yml and push.yml,
5+
# it is uses to check that the checked in copy of generated code
6+
# matches what the current submodule version of smithy-dafny generates.
7+
# This is important to ensure whenever someone changes the models
8+
# or needs to regenerate to pick up smithy-dafny improvements,
9+
# they don't have to deal with unpleasant surprises.
10+
#
11+
# 2. For workflows that check compatibility with other Dafny versions,
12+
# such as nightly_dafny.yml, it is necessary to regenerate the code
13+
# for that version of Dafny first.
14+
# This is ultimately because some of the code smithy-dafny generates
15+
# is tightly coupled to what code Dafny itself generates.
16+
# A concrete example is that Dafny 4.3 added TypeDescriptors
17+
# as parameters when constructing datatypes like Option and Result.
18+
#
19+
# This is why this is a composite action instead of a reusable workflow:
20+
# the latter executes in a separate runner environment,
21+
# but here we need to actually overwrite the generated code in place
22+
# so that subsequent steps can work correctly.
23+
#
24+
# This action assumes that the given version of Dafny and .NET 6.0.x
25+
# have already been set up, since they are used to format generated code.
26+
#
27+
# Note that recursively generating code doesn't currently work in this repo
28+
# with the version of the mpl pinned by the submodule,
29+
# because the SharedMakefileV2.mk in it doesn't work with newer versions of smithy-dafny.
30+
# Therefore by default we don't recursively regenerate code
31+
# (accomplished by setting the POLYMORPH_DEPENDENCIES environment variable to "").
32+
# If `update-and-regenerate-mpl` is true, we first pull the latest mpl,
33+
# which is necessary both for Makefile compatibility and so we can regenerate mpl code
34+
# for compatibility with newer versions of Dafny.
35+
#
36+
37+
name: "Polymorph code generation"
38+
description: "Regenerates code using smithy-dafny, and optionally checks that the result matches the checked in state"
39+
inputs:
40+
dafny:
41+
description: "The Dafny version to run"
42+
required: true
43+
type: string
44+
library:
45+
description: "Name of the library to regenerate code for"
46+
required: true
47+
type: string
48+
diff-generated-code:
49+
description: "Diff regenerated code against committed state"
50+
required: true
51+
type: boolean
52+
update-and-regenerate-mpl:
53+
description: "Locally update MPL to the tip of master and regenerate its code too"
54+
required: false
55+
default: false
56+
type: boolean
57+
runs:
58+
using: "composite"
59+
steps:
60+
- name: Update MPL submodule locally if requested
61+
if: inputs.update-and-regenerate-mpl == 'true'
62+
working-directory: submodules/MaterialProviders
63+
shell: bash
64+
run: |
65+
git checkout main
66+
git pull
67+
git submodule update --init --recursive
68+
69+
- name: Update top-level project.properties file in MPL
70+
if: inputs.update-and-regenerate-mpl == 'true'
71+
shell: bash
72+
working-directory: submodules/MaterialProviders
73+
run: |
74+
make generate_properties_file
75+
76+
# Update the project.properties file so that we pick up the right runtimes etc.,
77+
# in cases where inputs.dafny is different from the current value in that file.
78+
- name: Generate smithy-dafny-project.properties file
79+
if: inputs.update-and-regenerate-mpl == 'true'
80+
env:
81+
DAFNY_VERSION: ${{ inputs.dafny }}
82+
shell: bash
83+
run: |
84+
make generate_properties_file
85+
86+
- name: Update top-level project.properties file
87+
if: inputs.update-and-regenerate-mpl == 'true'
88+
shell: bash
89+
run: |
90+
awk -F= '!a[$1]++' smithy-dafny-project.properties project.properties > merged.properties
91+
mv merged.properties project.properties
92+
cat project.properties
93+
94+
- name: Don't regenerate dependencies unless requested
95+
id: dependencies
96+
shell: bash
97+
run: |
98+
echo "PROJECT_DEPENDENCIES=${{ inputs.update-and-regenerate-mpl != 'true' && 'PROJECT_DEPENDENCIES=' || '' }}" >> $GITHUB_OUTPUT
99+
100+
- name: Regenerate Dafny code using smithy-dafny
101+
# Unfortunately Dafny codegen doesn't work on Windows:
102+
# https://github.com/smithy-lang/smithy-dafny/issues/317
103+
if: runner.os != 'Windows'
104+
working-directory: ./${{ inputs.library }}
105+
shell: bash
106+
run: |
107+
make polymorph_dafny ${{ steps.dependencies.outputs.PROJECT_DEPENDENCIES }}
108+
109+
- name: Set up prettier in MPL
110+
if: inputs.update-and-regenerate-mpl == 'true'
111+
shell: bash
112+
# Annoyingly, prettier has to be installed in each library individually.
113+
# And this is only necessary or even possible if we've updated the mpl submodule.
114+
run: |
115+
make -C submodules/MaterialProviders/AwsCryptographyPrimitives setup_prettier
116+
make -C submodules/MaterialProviders/AwsCryptographicMaterialProviders setup_prettier
117+
make -C submodules/MaterialProviders/ComAmazonawsKms setup_prettier
118+
make -C submodules/MaterialProviders/ComAmazonawsDynamodb setup_prettier
119+
120+
- name: Regenerate Java code using smithy-dafny
121+
# npx seems to be unavailable on Windows GHA runners,
122+
# so we don't regenerate Java code on them either.
123+
if: runner.os != 'Windows'
124+
working-directory: ./${{ inputs.library }}
125+
shell: bash
126+
# smithy-dafny also formats generated code itself now,
127+
# so prettier is a necessary dependency.
128+
run: |
129+
make setup_prettier
130+
make polymorph_java ${{ steps.dependencies.outputs.PROJECT_DEPENDENCIES }}
131+
132+
- name: Regenerate .NET code using smithy-dafny
133+
working-directory: ./${{ inputs.library }}
134+
shell: bash
135+
run: |
136+
make polymorph_dotnet ${{ steps.dependencies.outputs.PROJECT_DEPENDENCIES }}
137+
138+
- name: Check regenerated code against commited code
139+
# Composite action inputs seem to not actually support booleans properly for some reason
140+
if: inputs.diff-generated-code == 'true'
141+
working-directory: ./${{ inputs.library }}
142+
shell: bash
143+
run: |
144+
make check_polymorph_diff

‎.github/workflows/ci_codegen.yml

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
# This workflow regenerates code using smithy-dafny and checks that the output matches what's checked in.
2+
name: Library Code Generation
3+
on:
4+
pull_request:
5+
push:
6+
branches:
7+
- main
8+
9+
jobs:
10+
code-generation:
11+
strategy:
12+
fail-fast: false
13+
matrix:
14+
library:
15+
[
16+
DynamoDbEncryption,
17+
TestVectors
18+
]
19+
# Note dotnet is only used for formatting generated code
20+
# in this workflow
21+
dotnet-version: ["6.0.x"]
22+
os: [ubuntu-latest]
23+
runs-on: ${{ matrix.os }}
24+
defaults:
25+
run:
26+
shell: bash
27+
env:
28+
DOTNET_CLI_TELEMETRY_OPTOUT: 1
29+
DOTNET_NOLOGO: 1
30+
steps:
31+
- name: Support longpaths
32+
run: |
33+
git config --global core.longpaths true
34+
35+
- uses: actions/checkout@v3
36+
with:
37+
submodules: recursive
38+
39+
# Only used to format generated code
40+
# and to translate version strings such as "nightly-latest"
41+
# to an actual DAFNY_VERSION.
42+
- name: Setup Dafny
43+
uses: dafny-lang/[email protected]
44+
with:
45+
dafny-version: 4.2.0
46+
47+
- name: Setup .NET Core SDK ${{ matrix.dotnet-version }}
48+
uses: actions/setup-dotnet@v3
49+
with:
50+
dotnet-version: ${{ matrix.dotnet-version }}
51+
52+
- uses: ./.github/actions/polymorph_codegen
53+
with:
54+
dafny: ${{ env.DAFNY_VERSION }}
55+
library: ${{ matrix.library }}
56+
diff-generated-code: true

‎.github/workflows/ci_test_java.yml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,15 @@ jobs:
5858
# A && B || C is the closest thing to an if .. then ... else ... or ?: expression the GitHub Actions syntax supports.
5959
dafny-version: ${{ (github.event_name == 'schedule' || inputs.nightly) && 'nightly-latest' || '4.2.0' }}
6060

61+
- name: Regenerate code using smithy-dafny if necessary
62+
if: ${{ inputs.nightly }}
63+
uses: ./.github/actions/polymorph_codegen
64+
with:
65+
dafny: ${{ env.DAFNY_VERSION }}
66+
library: ${{ matrix.library }}
67+
diff-generated-code: false
68+
update-and-regenerate-mpl: true
69+
6170
- name: Setup Java ${{ matrix.java-version }}
6271
uses: actions/setup-java@v4
6372
with:

‎.github/workflows/ci_test_net.yml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,15 @@ jobs:
6565
# A && B || C is the closest thing to an if .. then ... else ... or ?: expression the GitHub Actions syntax supports.
6666
dafny-version: ${{ (github.event_name == 'schedule' || inputs.nightly) && 'nightly-latest' || '4.2.0' }}
6767

68+
- name: Regenerate code using smithy-dafny if necessary
69+
if: ${{ inputs.nightly }}
70+
uses: ./.github/actions/polymorph_codegen
71+
with:
72+
dafny: ${{ env.DAFNY_VERSION }}
73+
library: ${{ matrix.library }}
74+
diff-generated-code: false
75+
update-and-regenerate-mpl: true
76+
6877
- name: Download Dependencies
6978
working-directory: ./${{ matrix.library }}
7079
run: make setup_net

‎.github/workflows/ci_verification.yml

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ jobs:
2828
# Don't run the nightly build on forks
2929
if: github.event_name != 'schedule' || github.repository_owner == 'aws'
3030
strategy:
31+
fail-fast: false
3132
matrix:
3233
# Break up verification between namespaces over multiple
3334
# actions to take advantage of parallelization
@@ -51,8 +52,17 @@ jobs:
5152
with:
5253
# A && B || C is the closest thing to an if .. then ... else ... or ?: expression the GitHub Actions syntax supports.
5354
dafny-version: ${{ (github.event_name == 'schedule' || inputs.nightly) && 'nightly-latest' || '4.2.0' }}
55+
56+
- name: Regenerate code using smithy-dafny if necessary
57+
if: ${{ inputs.nightly }}
58+
uses: ./.github/actions/polymorph_codegen
59+
with:
60+
dafny: ${{ env.DAFNY_VERSION }}
61+
library: DynamoDbEncryption
62+
diff-generated-code: false
63+
update-and-regenerate-mpl: true
5464

55-
- name: Verify ${{ matrix.library }} Dafny code
65+
- name: Verify ${{ matrix.service }} Dafny code
5666
shell: bash
5767
working-directory: ./DynamoDbEncryption
5868
run: |

‎DynamoDbEncryption/codegen-patches/DynamoDbEncryption/dotnet/dafny-4.2.0.patch

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
diff --git b/DynamoDbEncryption/runtimes/net/Generated/DynamoDbEncryption/TypeConversion.cs a/DynamoDbEncryption/runtimes/net/Generated/DynamoDbEncryption/TypeConversion.cs
2+
index 9a951767..5c0cee33 100644
3+
--- b/DynamoDbEncryption/runtimes/net/Generated/DynamoDbEncryption/TypeConversion.cs
4+
+++ a/DynamoDbEncryption/runtimes/net/Generated/DynamoDbEncryption/TypeConversion.cs
5+
@@ -7,6 +7,43 @@ namespace AWS.Cryptography.DbEncryptionSDK.DynamoDb
6+
{
7+
public static class TypeConversion
8+
{
9+
+ // BEGIN MANUAL EDIT
10+
+ public static AWS.Cryptography.KeyStore.KeyStore FromDafny_N3_aws__N12_cryptography__N15_dbEncryptionSdk__N8_dynamoDb__S17_KeyStoreReference(software.amazon.cryptography.keystore.internaldafny.types.IKeyStoreClient value)
11+
+ {
12+
+ if (value is software.amazon.cryptography.keystore.internaldafny.types.IKeyStoreClient dafnyValue)
13+
+ {
14+
+ return new AWS.Cryptography.KeyStore.KeyStore(dafnyValue);
15+
+ }
16+
+ throw new System.ArgumentException("Custom implementations of AWS.Cryptography.KeyStore.KeyStore are not supported yet");
17+
+ }
18+
+ public static software.amazon.cryptography.keystore.internaldafny.types.IKeyStoreClient ToDafny_N3_aws__N12_cryptography__N15_dbEncryptionSdk__N8_dynamoDb__S17_KeyStoreReference(AWS.Cryptography.KeyStore.KeyStore value)
19+
+ {
20+
+ if (value is AWS.Cryptography.KeyStore.KeyStore nativeValue)
21+
+ {
22+
+ return nativeValue.impl();
23+
+ }
24+
+ throw new System.ArgumentException("Custom implementations of AWS.Cryptography.KeyStore.KeyStore are not supported yet");
25+
+ }
26+
+ public static AWS.Cryptography.DbEncryptionSDK.DynamoDb.ILegacyDynamoDbEncryptor FromDafny_N3_aws__N12_cryptography__N15_dbEncryptionSdk__N8_dynamoDb__S32_LegacyDynamoDbEncryptorReference(software.amazon.cryptography.dbencryptionsdk.dynamodb.internaldafny.types.ILegacyDynamoDbEncryptor value)
27+
+ {
28+
+ if (value is NativeWrapper_LegacyDynamoDbEncryptor nativeWrapper) return nativeWrapper._impl;
29+
+ return new LegacyDynamoDbEncryptor(value);
30+
+
31+
+ }
32+
+ public static software.amazon.cryptography.dbencryptionsdk.dynamodb.internaldafny.types.ILegacyDynamoDbEncryptor ToDafny_N3_aws__N12_cryptography__N15_dbEncryptionSdk__N8_dynamoDb__S32_LegacyDynamoDbEncryptorReference(AWS.Cryptography.DbEncryptionSDK.DynamoDb.ILegacyDynamoDbEncryptor value)
33+
+ {
34+
+ switch (value)
35+
+ {
36+
+ case LegacyDynamoDbEncryptor valueWithImpl:
37+
+ return valueWithImpl._impl;
38+
+ case LegacyDynamoDbEncryptorBase nativeImpl:
39+
+ return new NativeWrapper_LegacyDynamoDbEncryptor(nativeImpl);
40+
+ default:
41+
+ throw new System.ArgumentException(
42+
+ "Custom implementations of LegacyDynamoDbEncryptor must extend LegacyDynamoDbEncryptorBase.");
43+
+ }
44+
+ }
45+
+ // END MANUAL EDIT
46+
public static AWS.Cryptography.DbEncryptionSDK.DynamoDb.BeaconKeySource FromDafny_N3_aws__N12_cryptography__N15_dbEncryptionSdk__N8_dynamoDb__S15_BeaconKeySource(software.amazon.cryptography.dbencryptionsdk.dynamodb.internaldafny.types._IBeaconKeySource value)
47+
{
48+
software.amazon.cryptography.dbencryptionsdk.dynamodb.internaldafny.types.BeaconKeySource concrete = (software.amazon.cryptography.dbencryptionsdk.dynamodb.internaldafny.types.BeaconKeySource)value;

‎DynamoDbEncryption/dafny/DynamoDbEncryption/Model/AwsCryptographyDbEncryptionSdkDynamoDbTypes.dfy

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -453,17 +453,17 @@ abstract module AbstractAwsCryptographyDbEncryptionSdkDynamoDbService
453453
import Operations : AbstractAwsCryptographyDbEncryptionSdkDynamoDbOperations
454454
function method DefaultDynamoDbEncryptionConfig(): DynamoDbEncryptionConfig
455455
method DynamoDbEncryption(config: DynamoDbEncryptionConfig := DefaultDynamoDbEncryptionConfig())
456-
returns (res: Result<IDynamoDbEncryptionClient, Error>)
456+
returns (res: Result<DynamoDbEncryptionClient, Error>)
457457
ensures res.Success? ==>
458458
&& fresh(res.value)
459459
&& fresh(res.value.Modifies)
460460
&& fresh(res.value.History)
461461
&& res.value.ValidState()
462462

463-
// Helper function for the benefit of native code to create a Success(client) without referring to Dafny internals
463+
// Helper functions for the benefit of native code to create a Success(client) without referring to Dafny internals
464464
function method CreateSuccessOfClient(client: IDynamoDbEncryptionClient): Result<IDynamoDbEncryptionClient, Error> {
465465
Success(client)
466-
} // Helper function for the benefit of native code to create a Failure(error) without referring to Dafny internals
466+
}
467467
function method CreateFailureOfError(error: Error): Result<IDynamoDbEncryptionClient, Error> {
468468
Failure(error)
469469
}

‎DynamoDbEncryption/dafny/DynamoDbEncryption/src/Beacon.dfy

Lines changed: 0 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -439,17 +439,4 @@ module BaseBeacon {
439439
else
440440
[HexChar(TruncateNibble(bytes[0] % 16, topBits))] + ToHexString(bytes[1..])
441441
}
442-
443-
lemma {:vcs_split_on_every_assert} CheckBytesToHex()
444-
ensures
445-
&& var bytes := [1,2,3,4,5,6,7,0xb7];
446-
&& BytesToHex(bytes, 1) == "1"
447-
&& BytesToHex(bytes, 2) == "3"
448-
&& BytesToHex(bytes, 3) == "7"
449-
&& BytesToHex(bytes, 4) == "7"
450-
&& BytesToHex(bytes, 5) == "17"
451-
&& BytesToHex(bytes, 6) == "37"
452-
&& BytesToHex(bytes, 7) == "37"
453-
&& BytesToHex(bytes, 8) == "b7"
454-
{}
455442
}

‎DynamoDbEncryption/dafny/DynamoDbEncryption/src/Index.dfy

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ module
2828
}
2929

3030
method DynamoDbEncryption(config: DynamoDbEncryptionConfig)
31-
returns (res: Result<IDynamoDbEncryptionClient, Error>)
31+
returns (res: Result<DynamoDbEncryptionClient, Error>)
3232
ensures res.Success? ==> res.value is DynamoDbEncryptionClient
3333
{
3434
var internalConfig := Operations.Config();

‎DynamoDbEncryption/dafny/DynamoDbEncryption/test/Beacon.dfy

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -739,4 +739,20 @@ module TestBaseBeacon {
739739
var src := GetLiteralSource([1,2,3,4,5], version);
740740
var bv :- expect C.ConvertVersionWithSource(FullTableConfig, version, src);
741741
}
742+
743+
method {:test} CheckBytesToHex()
744+
{
745+
var bytes := [1,2,3,4,5,6,7,0xb7];
746+
// These could be asserted instead,
747+
// but they get too expensive
748+
// on more recent versions of Dafny.
749+
expect BytesToHex(bytes, 1) == "1";
750+
expect BytesToHex(bytes, 2) == "3";
751+
expect BytesToHex(bytes, 3) == "7";
752+
expect BytesToHex(bytes, 4) == "7";
753+
expect BytesToHex(bytes, 5) == "17";
754+
expect BytesToHex(bytes, 6) == "37";
755+
expect BytesToHex(bytes, 7) == "37";
756+
expect BytesToHex(bytes, 8) == "b7";
757+
}
742758
}

‎DynamoDbEncryption/dafny/DynamoDbEncryption/test/BeaconTestFixtures.dfy

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -211,6 +211,7 @@ module BeaconTestFixtures {
211211
ensures fresh(output.client.Modifies)
212212
{
213213
var client :- expect Primitives.AtomicPrimitives();
214+
214215
var keyNameSet := set b <- version.standardBeacons :: b.name;
215216
var keyNames := SortedSets.ComputeSetToOrderedSequence2(keyNameSet, CharLess);
216217
var keys :- expect SI.GetHmacKeys(client, keyNames, keyNames, key);

0 commit comments

Comments
 (0)