API "items" cursor bug

If you fetch all items filtered by something like ‘parts’:‘defn’, then when you reach the end of items you will keep getting the same (non-empty) result with the same cursor.

If you don’t filter then the last response has no ‘cursor’ and you know to stop.

I suspect the ‘cursor’ is being set after the filtering, so if the last item is filtered out you can never finish.

I’m curious if a batch request has some method of handling this, but I’m kind of afraid to find out if the batch request also gets stuck in an infinite loop.

This might actually be quirkiness with the Google Datastore API cursors. I can’t find the documentation for the legacy version we’re using but I recall it saying that the cursor may or may not return null or undefined when the end of the query has been reached. I remember they went out of their way to make note of that in the old documentation that it can be unreliable.

https://cloud.google.com/datastore/docs/concepts/queries#limitations_of_cursors might shed some light on it, but these docs are for a newer version of the API. It makes you wonder when you read lines like: Cursors don't always work as expected with a query that uses an inequality filter or a sort order on a property with multiple values.

I don’t have a great answer for this other than perhaps shedding a little light. A workaround might be to just fetch all parts and then split them into arrays based on the part. It’s a bit inefficient if you’re only needing to use one part. The other slightly hackish workaround would be to check to make sure the returned array is not null and that the last id in it does not match the previous result.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.