Skip to content

Commit 1fa1e61

Browse files
lukealvoeirolukealvoeiro
authored
feat: support proto2 optional and default value fields (#1007)
### Description This PR aims to implement the features necessary to enable proto2 optional fields and support proto2 default values, requested by #973 The following changes are present in this PR: - Code changes to `main.ts` and `types.ts` that implement **optional fields**, as well as [**default values**](https://protobuf.dev/programming-guides/proto2/). - Creates two new options: `disableProto2Optionals` and `disableProto2DefaultValues`. By default these are set to false. - New integration tests for `proto3` and `proto2` syntax types, intended to catch any regressions - Modifications to existing integration tests - this mainly involves: - Updating message interfaces to use optional variables - Updating the createBaseMessage() function to use default values provided by the proto2 syntax. - The following two changes to the `encode` function of a message 1. When checking boolean default values, following the other default checks by checking if not equals instead of equals: ```ts // BEFORE message.truth === true // AFTER message.truth !== false ``` 2. When default checking Long types using the `forceLong=true` option, always check via a `.equals()` instead of `.isZero()`. This could be reverted, however, I find it to be cleaner; regardless of whether there is a default value provided or not this check will always work. ```ts // BEFORE !message.key.isZero() // AFTER !message.key.equals(Long.ZERO) ``` ### Outstanding work In a **subsequent future PRs**, we need to handle the following: - Updating to `ts-proto-descriptors` to correctly render fields as optional. This involves running changing the parameters for `protos/build.sh` to using the build code from `ts-protos`. I tried doing this as part of this PR, but the updated proto descriptors did not work well with ts-proto. Going to re-attempt this at a later date. - Fix the last of the default values: currently all but the Bytes type is handled correctly. I have added a `todo(proto2)` label around the places that need updating. ## Past Discussion Please ignore everything below, as it is out of date. I am keeping it within the PR description so below comments make sense to future readers. ##### Broken TypeScript in `src` > @lukealvoeiro: The code in `src/` is currently broken as a result of the `ts-proto-descriptors` update, which made a lot of the fields typically available in interfaces such as `ProtoFileDescriptor` optional. > > I'm planning on updating all the source code to use the optional syntax (not actually that hard, there are like 60 TS errors which I can probably knock out quickly), and then adding a new option that disables proto2 optional fields (e.g. `disableProto2Optionals=true`). This gives us better maintainability in the long run, brings us in line with proto2 conventions, and improves our parsing of proto files. It also allows customers who are used to ts-proto's existing output for proto2 files to continue to codegen the same TS output via the `disableProto2Optionals` flag. As I mentioned above, I tried do this but didn't get very far down this path. I think in general it makes sense to update `ts-proto-descriptors` to use the optional fields (and default values), however, it was harder than expected to get the updated descriptors working with `ts-proto` ##### File-specific context > @lukealvoeiro: One of the not-so-elegant aspects of this PR is visible in `main.ts`, where I am passing a parameter `isProto3File: boolean` around a lot. Wanted to open up a discussion as to whether it would be helpful to insert file-specific context such as `isProto3File` into the Context object. After we process each file, we could then overwrite this portion of the context with the next file's metadata. I implemented this change, folks can continue adding to the file context whenever appropriate. ### Testing performed - [x] Ran unit tests, changes to files are expected and compile / pass tests - [x] Ran on my own codebase that has proto2 and proto3 files and noticed no issues there either.
1 parent 51c3d0c commit 1fa1e61

File tree

64 files changed

+5195
-524
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

64 files changed

+5195
-524
lines changed

‎README.markdown

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -496,6 +496,10 @@ Generated code will be placed in the Gradle build directory.
496496

497497
- With `--ts_proto_opt=initializeFieldsAsUndefined=false`, all optional field initializers will be omited from the generated base instances.
498498

499+
- With `--ts_proto_opt=disableProto2Optionals=true`, all optional fields on proto2 files will not be set to be optional. Please note that this flag is primarily for preserving ts-proto's legacy handling of proto2 files, to avoid breaking changes, and as a result, it is not intended to be used moving forward.
500+
501+
- With `--ts_proto_opt=disableProto2DefaultValues=true`, all fields in proto2 files that specify a default value will not actually use that default value. Please note that this flag is primarily for preserving ts-proto's legacy handling of proto2 files, to avoid breaking changes, and as a result, it is not intended to be used moving forward.
502+
499503
- With `--ts_proto_opt=Mgoogle/protobuf/empty.proto=./google3/protobuf/empty`, ('M' means 'importMapping', similar to [protoc-gen-go](https://developers.google.com/protocol-buffers/docs/reference/go-generated#package)), the generated code import path for `./google/protobuf/empty.ts` will reflect the overridden value:
500504

501505
- `Mfoo/bar.proto=@myorg/some-lib` will map `foo/bar.proto` imports into `import ... from '@myorg/some-lib'`.

‎integration/before-after-request-streaming/google/protobuf/wrappers.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -442,7 +442,7 @@ function createBaseBoolValue(): BoolValue {
442442

443443
export const BoolValue = {
444444
encode(message: BoolValue, writer: _m0.Writer = _m0.Writer.create()): _m0.Writer {
445-
if (message.value === true) {
445+
if (message.value !== false) {
446446
writer.uint32(8).bool(message.value);
447447
}
448448
return writer;
@@ -477,7 +477,7 @@ export const BoolValue = {
477477

478478
toJSON(message: BoolValue): unknown {
479479
const obj: any = {};
480-
if (message.value === true) {
480+
if (message.value !== false) {
481481
obj.value = message.value;
482482
}
483483
return obj;

‎integration/bytes-node/google/protobuf/wrappers.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -442,7 +442,7 @@ function createBaseBoolValue(): BoolValue {
442442

443443
export const BoolValue = {
444444
encode(message: BoolValue, writer: _m0.Writer = _m0.Writer.create()): _m0.Writer {
445-
if (message.value === true) {
445+
if (message.value !== false) {
446446
writer.uint32(8).bool(message.value);
447447
}
448448
return writer;
@@ -477,7 +477,7 @@ export const BoolValue = {
477477

478478
toJSON(message: BoolValue): unknown {
479479
const obj: any = {};
480-
if (message.value === true) {
480+
if (message.value !== false) {
481481
obj.value = message.value;
482482
}
483483
return obj;

‎integration/codegen.ts

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@ import { generateFile, makeUtils } from "../src/main";
66
import { createTypeMap } from "../src/types";
77
import { generateIndexFiles } from "../src/utils";
88
import { getTsPoetOpts, optionsFromParameter } from "../src/options";
9-
import { Context } from "../src/context";
9+
import { BaseContext, createFileContext } from "../src/context";
1010
import { generateTypeRegistry } from "../src/generate-type-registry";
1111

1212
/**
@@ -37,8 +37,8 @@ async function generate(binFile: string, baseDir: string, parameter: string) {
3737
continue;
3838
}
3939
const utils = makeUtils(options);
40-
const ctx: Context = { options, typeMap, utils };
41-
const [path, code] = generateFile(ctx, file);
40+
const ctx: BaseContext = { options, typeMap, utils };
41+
const [path, code] = generateFile({ ...ctx, currentFile: createFileContext(file) }, file);
4242
const filePath = `${baseDir}/${path}`;
4343
const dirPath = parse(filePath).dir;
4444
await promisify(mkdir)(dirPath, { recursive: true }).catch(() => {});
@@ -47,7 +47,7 @@ async function generate(binFile: string, baseDir: string, parameter: string) {
4747

4848
if (options.outputTypeRegistry) {
4949
const utils = makeUtils(options);
50-
const ctx: Context = { options, typeMap, utils };
50+
const ctx: BaseContext = { options, typeMap, utils };
5151

5252
const path = "typeRegistry.ts";
5353
const code = generateTypeRegistry(ctx);

‎integration/emit-default-values-json/test.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -120,7 +120,7 @@ export const DefaultValuesTest = {
120120
if (message.long !== 0) {
121121
writer.uint32(32).int64(message.long);
122122
}
123-
if (message.truth === true) {
123+
if (message.truth !== false) {
124124
writer.uint32(40).bool(message.truth);
125125
}
126126
if (message.description !== "") {

‎integration/extensions/test.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -522,7 +522,7 @@ export const long: Extension<Long> = {
522522
packed: false,
523523
encode: (value: Long): Uint8Array[] => {
524524
const encoded: Uint8Array[] = [];
525-
if (value !== undefined && !value.isZero()) {
525+
if (value !== undefined && !value.equals(Long.ZERO)) {
526526
const writer = _m0.Writer.create();
527527
writer.int64(value);
528528
encoded.push(writer.finish());
@@ -542,7 +542,7 @@ export const fixed: Extension<Long> = {
542542
packed: false,
543543
encode: (value: Long): Uint8Array[] => {
544544
const encoded: Uint8Array[] = [];
545-
if (value !== undefined && !value.isZero()) {
545+
if (value !== undefined && !value.equals(Long.UZERO)) {
546546
const writer = _m0.Writer.create();
547547
writer.fixed64(value);
548548
encoded.push(writer.finish());

‎integration/global-this/global-this.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -204,7 +204,7 @@ function createBaseBoolean(): Boolean {
204204

205205
export const Boolean = {
206206
encode(message: Boolean, writer: _m0.Writer = _m0.Writer.create()): _m0.Writer {
207-
if (message.value === true) {
207+
if (message.value !== false) {
208208
writer.uint32(8).bool(message.value);
209209
}
210210
return writer;
@@ -239,7 +239,7 @@ export const Boolean = {
239239

240240
toJSON(message: Boolean): unknown {
241241
const obj: any = {};
242-
if (message.value === true) {
242+
if (message.value !== false) {
243243
obj.value = message.value;
244244
}
245245
return obj;

‎integration/grpc-js/google/protobuf/wrappers.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -442,7 +442,7 @@ function createBaseBoolValue(): BoolValue {
442442

443443
export const BoolValue = {
444444
encode(message: BoolValue, writer: _m0.Writer = _m0.Writer.create()): _m0.Writer {
445-
if (message.value === true) {
445+
if (message.value !== false) {
446446
writer.uint32(8).bool(message.value);
447447
}
448448
return writer;
@@ -477,7 +477,7 @@ export const BoolValue = {
477477

478478
toJSON(message: BoolValue): unknown {
479479
const obj: any = {};
480-
if (message.value === true) {
480+
if (message.value !== false) {
481481
obj.value = message.value;
482482
}
483483
return obj;

‎integration/grpc-web/google/protobuf/wrappers.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -442,7 +442,7 @@ function createBaseBoolValue(): BoolValue {
442442

443443
export const BoolValue = {
444444
encode(message: BoolValue, writer: _m0.Writer = _m0.Writer.create()): _m0.Writer {
445-
if (message.value === true) {
445+
if (message.value !== false) {
446446
writer.uint32(8).bool(message.value);
447447
}
448448
return writer;
@@ -477,7 +477,7 @@ export const BoolValue = {
477477

478478
toJSON(message: BoolValue): unknown {
479479
const obj: any = {};
480-
if (message.value === true) {
480+
if (message.value !== false) {
481481
obj.value = message.value;
482482
}
483483
return obj;

‎integration/map-long-optional/test.ts

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -129,10 +129,10 @@ function createBaseMapBigInt_MapEntry(): MapBigInt_MapEntry {
129129

130130
export const MapBigInt_MapEntry = {
131131
encode(message: MapBigInt_MapEntry, writer: _m0.Writer = _m0.Writer.create()): _m0.Writer {
132-
if (!message.key.isZero()) {
132+
if (!message.key.equals(Long.UZERO)) {
133133
writer.uint32(9).fixed64(message.key);
134134
}
135-
if (!message.value.isZero()) {
135+
if (!message.value.equals(Long.ZERO)) {
136136
writer.uint32(16).int64(message.value);
137137
}
138138
if (message._unknownFields !== undefined) {
@@ -204,10 +204,10 @@ export const MapBigInt_MapEntry = {
204204

205205
toJSON(message: MapBigInt_MapEntry): unknown {
206206
const obj: any = {};
207-
if (!message.key.isZero()) {
207+
if (!message.key.equals(Long.UZERO)) {
208208
obj.key = (message.key || Long.UZERO).toString();
209209
}
210-
if (!message.value.isZero()) {
210+
if (!message.value.equals(Long.ZERO)) {
211211
obj.value = (message.value || Long.ZERO).toString();
212212
}
213213
return obj;

‎integration/meta-typings/google/protobuf/wrappers.ts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -317,7 +317,7 @@ function createBaseBoolValue(): BoolValue {
317317

318318
export const BoolValue = {
319319
encode(message: BoolValue, writer: _m0.Writer = _m0.Writer.create()): _m0.Writer {
320-
if (message.value === true) {
320+
if (message.value !== false) {
321321
writer.uint32(8).bool(message.value);
322322
}
323323
return writer;

‎integration/nice-grpc/google/protobuf/wrappers.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -442,7 +442,7 @@ function createBaseBoolValue(): BoolValue {
442442

443443
export const BoolValue = {
444444
encode(message: BoolValue, writer: _m0.Writer = _m0.Writer.create()): _m0.Writer {
445-
if (message.value === true) {
445+
if (message.value !== false) {
446446
writer.uint32(8).bool(message.value);
447447
}
448448
return writer;
@@ -477,7 +477,7 @@ export const BoolValue = {
477477

478478
toJSON(message: BoolValue): unknown {
479479
const obj: any = {};
480-
if (message.value === true) {
480+
if (message.value !== false) {
481481
obj.value = message.value;
482482
}
483483
return obj;

‎integration/omit-optionals/simple.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ function createBaseTestMessage(): TestMessage {
1414

1515
export const TestMessage = {
1616
encode(message: TestMessage, writer: _m0.Writer = _m0.Writer.create()): _m0.Writer {
17-
if (message.field1 === true) {
17+
if (message.field1 !== false) {
1818
writer.uint32(8).bool(message.field1);
1919
}
2020
if (message.field2 !== undefined) {
@@ -62,7 +62,7 @@ export const TestMessage = {
6262

6363
toJSON(message: TestMessage): unknown {
6464
const obj: any = {};
65-
if (message.field1 === true) {
65+
if (message.field1 !== false) {
6666
obj.field1 = message.field1;
6767
}
6868
if (message.field2 !== undefined) {

‎integration/optional-long/google/protobuf/timestamp.ts

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,7 @@ function createBaseTimestamp(): Timestamp {
119119

120120
export const Timestamp = {
121121
encode(message: Timestamp, writer: _m0.Writer = _m0.Writer.create()): _m0.Writer {
122-
if (message.seconds !== undefined && !message.seconds.isZero()) {
122+
if (message.seconds !== undefined && !message.seconds.equals(Long.ZERO)) {
123123
writer.uint32(8).int64(message.seconds);
124124
}
125125
if (message.nanos !== undefined && message.nanos !== 0) {
@@ -167,7 +167,7 @@ export const Timestamp = {
167167

168168
toJSON(message: Timestamp): unknown {
169169
const obj: any = {};
170-
if (message.seconds !== undefined && !message.seconds.isZero()) {
170+
if (message.seconds !== undefined && !message.seconds.equals(Long.ZERO)) {
171171
obj.seconds = (message.seconds || Long.ZERO).toString();
172172
}
173173
if (message.nanos !== undefined && message.nanos !== 0) {

0 commit comments

Comments
 (0)