Document iterator

Document iterator via GET and POST are identical, only the way parameters are passed differ. The document iterator allows to iterate over all document IDs and documents in the entire index, forward or backward. It enables efficient sequential access to every document, even in very large indexes, without running a search. Paging through the index works without collecting document IDs to Min-heap in size-limited RAM first. The iterator guarantees that only valid document IDs are returned, even though document IDs are not strictly continuous. Document IDs can also be fetched in batches, reducing round trips and significantly improving performance, especially when using the REST API. Typical use cases include index export, conversion, analytics, audits, and inspection. Explanation of "eventually continuous" docid: In SeekStorm, document IDs become continuous over time. In a multi-sharded index, each shard maintains its own document ID space. Because documents are distributed across shards in a non-deterministic, load-dependent way, shard-local document IDs advance at different rates. When these are mapped to global document IDs, temporary gaps can appear. As a result, simply iterating from 0 to the total document count may encounter invalid IDs near the end. The Document Iterator abstracts this complexity and reliably returns only valid document IDs.

Parameters

• docid=None, take>0: skip first s document IDs, then take next t document IDs of an index.
• docid=None, take<0: skip last s document IDs, then take previous t document IDs of an index.
• docid=Some, take>0: skip next s document IDs, then take next t document IDs of an index, relative to a given document ID, with end-of-index indicator.
• docid=Some, take<0: skip previous s document IDs, then take previous t document IDs, relative to a given document ID, with start-of-index indicator.
• take=0: does not make sense, that defies the purpose of get_iterator.
• The sign of take indicates the direction of iteration: positive take for forward iteration, negative take for backward iteration.
• The skip parameter is always positive, indicating the number of document IDs to skip before taking document IDs. The skip direction is determined by the sign of take too.
• include_document: if true, the documents are also retrieved along with their document IDs. Next page: take last docid from previous result set, skip=1, take=+page_size Previous page: take first docid from previous result set, skip=1, take=-page_size Returns an IteratorResult, consisting of the number of actually skipped document IDs, and a list of taken document IDs and documents, sorted ascending). Detect end/begin of index during iteration: if returned vec.len() < requested take || if returned skip <requested skip