Feat: Check freshness for mixed external & sqlmesh models

[VaggelisD]

Follow up PR on model freshness (Previous PR)

This PR enables mixed SQLMesh & external (source) model freshness, e.g:

A: External model 
B: SELECT * FROM A
C: SELECT * FROM A, B

Model C will be evaluated if its upstream models are fresh, which is determined by:

• C's last_altered_ts compared to external model's (A) INFORMATION_SCHEMA metadata
• The existence (or not) of parent intervals for SQLMesh models (B), signalling if any model will be evaluated as part of this run

A new freshness-specific test file is added to break up the original test in various scenarios.

[VaggelisD]

TODO: Reinstate before merging

[izeigerman]

I don't think this is the right place to construct this. The missing intervals will change as we apply signal checks to snapshots. If you set them once here, the parent signals will not be reflected.

[VaggelisD]

Good call.

I guess our best choice would be to check snapshot_batches then (same loop) which applies the signal to filter out the unready intervals.

So, for each snapshot in the DAG, its ExecutionContext::parent_intervals list will be extended by the snapshot_batches[parent], provided the parent snapshot is NOT external.

[izeigerman]

yes, this sounds correct, assuming I understood the idea correctly

[VaggelisD]

Implemented in 9a8af2e

[izeigerman]

Are you sure that snapshot_intervals reflects updated intervals after signals were applied to parents. If I'm reading the code correctly, snapshot_intervals remain unchanged and only snapshot_batches reflect signal outcomes.

[izeigerman]

Sorry man, I'm still confused. external_parents contains external tables that were not registered as external models. Both upstream_parent_snapshots and context.parent_intervals exclude registered external models. So at which point do we check the last modified timestamp for tables that are registered external models?

[izeigerman]

I believe external_models should include both:

snapshot.node.depends_on - {p.name for p in upstream_parent_snapshots}

AND

{p.name for p in parent_snapshots if p.is_external}

am I missing something?