dtml-in batching accesses the complete list
#75
Comments
|
The behavior is caused by the list conversion in " ".It is there to support iterators in addition to true sequences but of course it destroys lazy access to a sequence (highly wished for batching). I do not yet know how to distinguish safely between a true sequence (not requiring a list conversion) and something requiring a list conversion (e.g. a general iterator). |
|
For what I see, there are a lot of places where sequence gets unpacked: to-list conversions and key-lookup access: List conversions: Key lookups: Key lookups I think are unnecessary because all the try-execeps blocks could be rewritten as simple if-then-else checks on start/end values. List conversions perhaps could be refactored to convert in list only the firsts "batch_size * Page number" elements. my 2 cent |
|
I found out that not sequence and this will be False for general iterators even if the iterator does not give any values. The list conversion comes too late to remedy this.
|
|
The original implementation tried hard to avoid the computation of the sequence length (by e.g. using I propose to retain the support for lazy batching as follows: |
|
Alessandro Ceglie wrote at 2024-9-27 03:12 -0700:
For what I see, there are a lot of places where sequence gets unpacked: to-list conversions and key-lookup access:
List conversions:
- https://github.com/zopefoundation/DocumentTemplate/blob/9496ce63234a849961ef7bcd85890632bb14bf3d/src/DocumentTemplate/DT_InSV.py#L41
- https://github.com/zopefoundation/DocumentTemplate/blob/9496ce63234a849961ef7bcd85890632bb14bf3d/src/DocumentTemplate/DT_In.py#L478
I propose the following approach:
We check (using a heuristics) whether the original sequence
supports "key access" and if it does we use it directly.
If it does not we use its list conversion.
Unfortunately, checking for "key access" is not trivial.
What we actually want to check for is sequence subscription, i.e.
we should look for `__getitem__`; but `__getitem__` is used for
mapping subscription, too.
Therefore, I tend to the following heuristics:
If the original sequence has `__len__` and `__getitem__` (potentially
a sequence) but is not an instance of `collections.abs.Mapping` and has neither
`keys` nor `get` (likely not a `Mapping`), then we use it directly.
An alternativ to a list conversion could be the use of
a lazy `iter2list` wrapper. It would ensure "key access", by
iteration and caching of the obtained values.
Key lookups:
If we have guaranteed that the used sequence does support "key lookup",
they no longer provide problems.
|
DocumentTemplate 4.0"zopefoundation/Products.ZCatalog#155" reports that
Products.ZCatalog'smanage_catalogViewtakes excessive times for large catalogs."zopefoundation/Products.ZCatalog#155 (comment)" contains corresponding profile results. They suggest that
dtml-inaccesses each element in the list even when called with batching parameters.The text was updated successfully, but these errors were encountered: