-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Description
Is this a new bug in dbt-core?
- I believe this is a new bug in dbt-core
- I have searched the existing issues, and I could not find an existing issue for this bug
Current Behavior
I have tried all combinations of old and new source configurations but am unable to get the loaded_at_query for source freshness to be honored on a freshness run when the source is a databricks table.
We are using the extended version of dbt cloud.
The debug logs show an event 'ROLLBACK' before the sql is compiled, the compiled sql for the freshness snapshot that gets used is the default Databricks sql statement: 'select table from system.information_schema.tables'
I have followed these docs following the format for setting the loaded_at_value in the model yaml for the source and they do not work: https://docs.getdbt.com/reference/resource-properties/freshness
We do not have a structure where we can set it in the project yaml.
I have tried the below query as the sql and also tried replacing target with the full source name like database.schema.table; """ loaded_at_query: 'select max(cast(generated_at AS timestamp)) from {{ target }}' """
The query is required because the source table comes from the elementary package and is a string, where the generated_at column is used as the loaded_at_field value it is respected, but an error is thrown because a timestamp data type is expected.
I have tried entering the loaded_at_query value inside and outside of the config block, both in the top level values of the source yaml and inside the table section of the source yaml. (outside the config block fails parsing on both attempts).
Though inside a config block it gets parsed successfully, it does not get respected on the freshness evaluation.
Expected Behavior
loaded_at_value gets respected for calculating freshness on the databricks source table.
Steps To Reproduce
- create a project which uses databricks as the target database
- configure a source as detailed above, using any valid location to define the loaded_at_query, also set the freshness values
- run a freshness job on the source
Relevant log output
Environment
- OS:
- Python:
- dbt: Extended track(date of error 2.9.2025) https://docs.getdbt.com/docs/dbt-versions/compatible-track-changelog
Which database adapter are you using with dbt?
No response
Additional Context
No response