Skip to content

Generate copyWithCompanion on data classes #3022

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
May 24, 2024

Conversation

gnprice
Copy link
Contributor

@gnprice gnprice commented May 24, 2024

The companion class makes an extremely useful way to bundle up a set of changes to make to a record. This copyWithCompanion method makes it possible to apply those changes directly to a record as represented in Dart, as an instance of the data class.

For example, it can be used by a wrapper that keeps a write-through cache in Dart for a given database table. Here's slightly simplified from a real example we have in Zulip:

/// Update an account in the store, returning the new version.
///
/// The account with the given account ID will be updated.
/// It must already exist in the store.
Future<Account> updateAccount(int accountId, AccountsCompanion data) async {
  assert(!data.id.present);
  await (_db.update(_db.accounts)
    ..where((a) => a.id.equals(accountId))
  ).write(data);
  final result = _accounts.update(accountId,
    (value) => value.copyWithCompanion(data));
  notifyListeners();
  return result;
}

It's possible to write such a method by hand, of course (in an extension), or to call copyWith directly. But it needs a line for each column of the table, which makes either of those error-prone: in particular, because copyWith naturally has its parameters all optional, it would be very easy for someone adding a new column to overlook the need to add a line to this method, and then updates to the new column would just silently get dropped. So this is a case where there's a large benefit to generating the code.

The companion class makes an extremely useful way to bundle up a set
of changes to make to a record.  This copyWithCompanion method makes
it possible to apply those changes directly to a record as represented
in Dart, as an instance of the data class.

For example, it can be used by a wrapper that keeps a write-through
cache in Dart for a given database table.  Here's slightly simplified
from a real example we have in Zulip:

    /// Update an account in the store, returning the new version.
    ///
    /// The account with the given account ID will be updated.
    /// It must already exist in the store.
    Future<Account> updateAccount(int accountId, AccountsCompanion data) async {
      assert(!data.id.present);
      await (_db.update(_db.accounts)
        ..where((a) => a.id.equals(accountId))
      ).write(data);
      final result = _accounts.update(accountId,
        (value) => value.copyWithCompanion(data));
      notifyListeners();
      return result;
    }

It's possible to write such a method by hand, of course (in an
extension), or to call copyWith directly.  But it needs a line for
each column of the table, which makes either of those error-prone:
in particular, because copyWith naturally has its parameters all
optional, it would be very easy for someone adding a new column to
overlook the need to add a line to this method, and then updates to
the new column would just silently get dropped.  So this is a case
where there's a large benefit to generating the code.
@gnprice
Copy link
Contributor Author

gnprice commented May 24, 2024

CI is failing, with a number of failures that I don't reproduce locally and appear unrelated to the PR.

For example, it has:

2024-05-24T02:04:45.3463056Z ##[group]❌ loading test/serialization_test.dart (failed)
2024-05-24T02:04:45.3463879Z Failed to load "test/serialization_test.dart":
2024-05-24T02:04:45.3466028Z test/generated/todos.g.dart:140:20: Error: Required named parameter 'descriptionInUpperCase' must be provided.
2024-05-24T02:04:45.3467141Z     return Category(
2024-05-24T02:04:45.3467599Z                    ^
2024-05-24T02:04:45.3468679Z test/generated/todos.g.dart:112:9: Context: Found this candidate, but the arguments don't match.
2024-05-24T02:04:45.3469657Z   const Category(
2024-05-24T02:04:45.3470054Z         ^^^^^^^^
2024-05-24T02:04:45.3470907Z package:test_core/src/runner/vm/platform.dart 242:7   VMPlatform._compileToKernel
2024-05-24T02:04:45.3471859Z ===== asynchronous gap ===========================
2024-05-24T02:04:45.3473398Z package:test_core/src/runner/vm/platform.dart 220:13  VMPlatform._spawnIsolate
2024-05-24T02:04:45.3475370Z ===== asynchronous gap ===========================
2024-05-24T02:04:45.3477008Z package:test_core/src/runner/vm/platform.dart 75:19   VMPlatform.load
2024-05-24T02:04:45.3478557Z ===== asynchronous gap ===========================
2024-05-24T02:04:45.3480072Z package:test_core/src/runner/loader.dart 219:27       Loader.loadFile.<fn>
2024-05-24T02:04:45.3481249Z ===== asynchronous gap ===========================
2024-05-24T02:04:45.3482393Z package:test_core/src/runner/load_suite.dart 98:19    new LoadSuite.<fn>.<fn>
2024-05-24T02:04:45.3483743Z ##[endgroup]

But that test file passes for me:

$ /usr/bin/dart test test/serialization_test.dart 
Building package executable... 
Built test:test.
00:00 +8: All tests passed!                                                                             

It seems like there's a mismatch somewhere over a parameter named descriptionInUpperCase, but I don't think my PR's changes interact with that.

@dickermoshe
Copy link
Collaborator

Something funny is going on with the entire package. It seems some dependencies were changes on upstream packages and things are kinda broken. I'm hoping Simone can figure it out. I've been pulling my hair out over this.

@dickermoshe
Copy link
Collaborator

dickermoshe commented May 24, 2024

Thank you for the contribution

Could you please clarify what you're adding here, it looks like a wayo combine Companion objects, right?
You've only posted a vague "Before" picture, with no "After" picture showing what here is being added & the generated code hasn't been re-built with melos build, so I've got nothing to look at either.

It would be helpful if you could put forth a proposal with an "Issue" and a "Solution". Would make this review much easier.

(It doesn't seem Value.absent would be handled correctly in the code you've posted above, although it's not really clear so, 🤷)

@simolus3
Copy link
Owner

It seems some dependencies were changes on upstream packages and things are kinda broken

Did you see any particular failures about this? We are caching dependencies across CI because that speeds things up, but we're re-running pub get across packages for each run. I don't think the cache is causing this failure, but I'm happy to take a deeper look if you see more suspicious CI things happening.

My best guess is that this might be because of the build_verify test, which runs build_runner build --delete-conflicting-outputs as a test and then verifies that nothing has changed. If we're doing that while the other tests are currently loading, we might be running into race conditions where the generated code is in an inconsistent state. Since we're running the build before tests in the CI, I've changed the test in f3487f3 to only to the diff. Let's see if that fixes the problem.

Copy link
Owner

@simolus3 simolus3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that re-running the build at least in the drift/ package would be helpful :) Then we can also have a small unit test calling this method on a generated data class/companion part as part of drift/test/database/data_class_test.dart.

This is definitely useful though and I agree with the changed, I just wonder if copyFromCompanion could be a clearer name.

@gnprice
Copy link
Contributor Author

gnprice commented May 24, 2024

My best guess is that this might be because of the build_verify test, which runs build_runner build --delete-conflicting-outputs as a test and then verifies that nothing has changed. If we're doing that while the other tests are currently loading, we might be running into race conditions where the generated code is in an inconsistent state. Since we're running the build before tests in the CI, I've changed the test in f3487f3 to only to the diff. Let's see if that fixes the problem.

Interesting, that makes sense! Thanks for the fix.

I'll start by just rebasing (or merging? from the Git history, it looks like your preference is to merge, so I'll do that), to rerun the CI with just that change, in order to try to confirm that that was the issue while I work on those other changes.

@gnprice
Copy link
Contributor Author

gnprice commented May 24, 2024

I just wonder if copyFromCompanion could be a clearer name.

Yeah, that would be a very reasonable name — definitely "copy from companion" reads better as a standalone phrase than "copy with companion" does.

I went for this name with the thought that it's very much like the existing (and conventional across Dart libraries generally) copyWith method; it basically does the exact same job as copyWith, just taking its arguments in a different format. So it seems nice to make the name similar so as to draw that connection.

Hmm, thinking more, I guess there's one way in which copyFromCompanion would potentially be misleading. A key part of the meaning of this method, like copyWith, is that it copies the receiver — it doesn't mutate it, but rather makes a new object which is a copy of the existing one. And then it makes that copy "with" some modifications, specified by the arguments. If the name were to read "copy from companion", then that would suggest a method that copies from the companion to… well, the most obvious potential destination might be the receiver itself, meaning mutating it.

So possibly copyWithCompanion is clearer after all, even though it reads a bit awkwardly. It really means "copy (this) with (modifications specified by a) companion".

@gnprice
Copy link
Contributor Author

gnprice commented May 24, 2024

Since we're running the build before tests in the CI, I've changed the test in f3487f3 to only to the diff. Let's see if that fixes the problem.

Well, it looks like the CI result after that change was the same. So that change makes sense, but doesn't seem to have fixed that problem on its own.

Anyway, clearly the generated files should get updated in any case 🙂, so I'll see about doing that and we'll find out if that was indeed the trigger for these other failures.

Ran the command:

    dart run build_runner build --delete-conflicting-outputs
@gnprice
Copy link
Contributor Author

gnprice commented May 24, 2024

Hmm, interesting! All those failures were in fact pointing to a bug in my changes — involving the interaction with a feature I wasn't aware of.

When the error says:

  test/generated/todos.g.dart:140:20: Error: Required named parameter 'descriptionInUpperCase' must be provided.
      return Category(

that offending Category constructor call site is one in a generated copyWithCompanion implementation. What's happening is that descriptionInUpperCase is, in that example Category class / Categories table, a generated column:

  TextColumn get descriptionInUpperCase =>
      text().generatedAs(description.upper())();

and that causes it to not be represented in companion classes.

After reading a bit more code, it looks like there isn't a way to compute what the value of that field should be, short of making a SQL query. (Fundamentally, doing so would require attempting to reimplement SQL expressions in Dart, which would be audacious; and I don't see anything that attempts that.) So perhaps the best thing here is that copyWithCompanion should just not be emitted when there are generated columns, because in that case a companion doesn't carry enough information to do the "copy-with" job.

I guess possibly the ideal answer is that copyWithCompanion would take additional, named, parameters for any generated columns. That way if there's one generated column and 12 other columns, one still gets the benefits of it for the other 12. I think what I'll try is to go for that if it turns out to be almost as easy as just skipping those classes/tables, and otherwise fall back to just skipping, leaving the fancier version as future work for when someone runs into that case and wants to add it.

@gnprice
Copy link
Contributor Author

gnprice commented May 24, 2024

OK, all tests are now passing! Thanks for your patience.

I skipped emitting this method when there are generated columns; updated the generated files in drift/; and also added one of these:

Then we can also have a small unit test calling this method on a generated data class/companion part as part of drift/test/database/data_class_test.dart.

Copy link
Owner

@simolus3 simolus3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair point about the name, I agree 👍

LGTM let's just check the dataClassToCompanions condition (if it's required we can leave it of course it just stood out).

@gnprice
Copy link
Contributor Author

gnprice commented May 24, 2024

Could you please clarify what you're adding here, it looks like a wayo combine Companion objects, right? You've only posted a vague "Before" picture, with no "After" picture showing what here is being added & the generated code hasn't been re-built with melos build, so I've got nothing to look at either.

Sure. The example call site in the PR description is actually the "After" picture as well as part of the "Before":

/// Update an account in the store, returning the new version.
///
/// The account with the given account ID will be updated.
/// It must already exist in the store.
Future<Account> updateAccount(int accountId, AccountsCompanion data) async {
  assert(!data.id.present);
  await (_db.update(_db.accounts)
    ..where((a) => a.id.equals(accountId))
  ).write(data);
  final result = _accounts.update(accountId,
    (value) => value.copyWithCompanion(data));
  notifyListeners();
  return result;
}

In the live code where I'm using a version of that example already, it's made possible by having written the method by hand in an extension, as I mentioned. Here's what that looks like:

extension AccountExtension on Account {
  Account copyWithCompanion(AccountsCompanion data) { // TODO(drift): generate this
    return Account(
      id: data.id.present ? data.id.value : id,
      realmUrl: data.realmUrl.present ? data.realmUrl.value : realmUrl,
      userId: data.userId.present ? data.userId.value : userId,
      email: data.email.present ? data.email.value : email,
      apiKey: data.apiKey.present ? data.apiKey.value : apiKey,
      zulipVersion: data.zulipVersion.present ? data.zulipVersion.value : zulipVersion,
      zulipMergeBase: data.zulipMergeBase.present ? data.zulipMergeBase.value : zulipMergeBase,
      zulipFeatureLevel: data.zulipFeatureLevel.present ? data.zulipFeatureLevel.value : zulipFeatureLevel,
      ackedPushToken: data.ackedPushToken.present ? data.ackedPushToken.value : ackedPushToken,
    );
  }
}

So the "Before" consists of that hand-written copyWithCompanion, plus the call site above.

There's an alternative "Before" option, which is to call copyWith directly as I mentioned. That would look like this:

  /// Update an account in the store, returning the new version.
  ///
  /// The account with the given account ID will be updated.
  /// It must already exist in the store.
  ///
  /// Fields that are given will be updated,
  /// and fields not given will be left unmodified.
  ///
  /// The `realmUrl` and `userId` fields should never change on an account,
  /// and therefore are not accepted as parameters.
  Future<Account> updateAccount(int accountId, {
    String? email,
    String? apiKey,
    String? zulipVersion,
    Value<String?> zulipMergeBase = const Value.absent(),
    int? zulipFeatureLevel,
    Value<String?> ackedPushToken = const Value.absent(),
  }) async {
    assert(_accounts.containsKey(accountId));
    await doUpdateAccount(accountId, AccountsCompanion(
      email: Value.absentIfNull(email),
      apiKey: Value.absentIfNull(apiKey),
      zulipVersion: Value.absentIfNull(zulipVersion),
      zulipMergeBase: zulipMergeBase,
      zulipFeatureLevel: Value.absentIfNull(zulipFeatureLevel),
      ackedPushToken: ackedPushToken,
    ));
    final result = _accounts.update(accountId, (value) => value.copyWith(
      email: email,
      apiKey: apiKey,
      zulipVersion: zulipVersion,
      zulipMergeBase: zulipMergeBase,
      zulipFeatureLevel: zulipFeatureLevel,
      ackedPushToken: ackedPushToken,
    ));
    notifyListeners();
    return result;
  }

As you can see, that's pretty repetitive. And more importantly, it's error-prone — because both the AccountsCompanion constructor and copyWith have all their parameters optional, when adding a column to the table it would be very easy to forget to update the calls to either or both of them in that updateAccount method, causing a bug.

@simolus3 simolus3 merged commit 51d2e8f into simolus3:develop May 24, 2024
10 checks passed
@simolus3
Copy link
Owner

Thanks for the contribution!

PIG208 added a commit to PIG208/zulip-flutter that referenced this pull request Dec 16, 2024
This has been addressed upstream in
  simolus3/drift#3022

Signed-off-by: Zixuan James Li <[email protected]>
PIG208 added a commit to PIG208/zulip-flutter that referenced this pull request Dec 16, 2024
This has been addressed in Greg's upstream PR:
  simolus3/drift#3022

Signed-off-by: Zixuan James Li <[email protected]>
PIG208 added a commit to PIG208/zulip-flutter that referenced this pull request Dec 16, 2024
This has been addressed in Greg's upstream PR:
  simolus3/drift#3022

Signed-off-by: Zixuan James Li <[email protected]>
PIG208 added a commit to PIG208/zulip-flutter that referenced this pull request Dec 17, 2024
This has been addressed in Greg's upstream PR:
  https://git hub.com/simolus3/drift/pull/3022

Signed-off-by: Zixuan James Li <[email protected]>
gnprice pushed a commit to PIG208/zulip-flutter that referenced this pull request Dec 19, 2024
This has been addressed in Greg's upstream PR:
  simolus3/drift#3022

which was pulled in by the Drift upgrade in 8b564e4 (zulip#1117).

Signed-off-by: Zixuan James Li <[email protected]>
@gnprice gnprice deleted the pr-copywithcompanion branch May 8, 2025 19:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants