Automatic Cache Refresh with iceberg.metadata-cache.enabled in Trino

[byunks]

Hello Trino Community,

I have a question regarding the metadata caching functionality in Iceberg. I am planning to use Trino with the iceberg.metadata-cache.enabled option set to true, which I understand helps to cache Iceberg table metadata and improve query performance. However, I am curious about how I can ensure that the cache is automatically refreshed when the underlying Iceberg table metadata changes.

Specifically:

1. Is there an automatic mechanism in Trino that invalidates or refreshes the metadata cache when changes occur in the Iceberg table schema or data?
2. If not, are there any recommended methods or configurations to manually control or trigger cache invalidation at set intervals (such as a TTL-like feature)?
3. If manual cache invalidation is required, what are the best practices for ensuring that queries always reflect the latest metadata without needing to restart the Trino server?

Any guidance or suggestions would be greatly appreciated. Thank you!

Best regards,
byunks

[raunaqmorarka]

This cache is based on iceberg metadata files which are guaranteed to be immutable by the iceberg spec.
So there is no real need for a refresh, as we never deal with stale metadata.
There are some configurables available at io.trino.filesystem.memory.MemoryFileSystemCacheConfig (undocumented at the moment) which can control TTL, max size and max content length for this cache.
cc: @mosabua

[byunks]

@raunaqmorarka

Hello,

Regarding the response, you mentioned that the cache is based on Iceberg metadata files, which are guaranteed to be immutable by the Iceberg spec.

I would like to understand this in more detail. When there are changes to the Iceberg table spec (such as adding or modifying columns, or changing sort, Z-order, partition, etc.), new metadata files are created. Could you clarify what is meant by "immutable" in this context when the spec of the Iceberg table can change and generate new metadata files?

Thank you for your explanation.

[raunaqmorarka]

What I meant is that there are no in-place updates to iceberg metadata files, we only add new metadata or delete old metadata that is no longer referenced. So the cache will fetch new metadata as needed and never serve stale metadata. This makes it unnecessary to manually refresh or invalidate the cache.

[byunks]

@raunaqmorarka
So, does this mean that when the table is updated with a new spec in Iceberg, a new metadata file is created and referenced, which is then automatically cached, while the old metadata file is automatically removed? Also, could you point me to the relevant code that handles this behavior?