[SOLVED] Evm API - Cursor Pagination

#1

I have transitioned from getContractEvents to getContractLogs and face the problem of cursor pagination.

I understand that the cursor pagination is more flexible and sustainable for large datasets, but is it possible to maintain both parameters? Meaning to also keep the offset based approach.

But I’d like to execute parallelized requests.

Basically, I first get the total count of elements in a block range and then execute all pages as different requests under my own rate limitations according to my subscription for optimal usage. This parallelisation and optimal usage of the requests is now no longer possible with the sequential cursor implementation.

Am I missing something or is it possible to somehow use offset with getContractLogs instead?

Thanks for any insights!

#2

It depends on what other parameters you can use for that endpoint. If you can pass a block range that you can still do concurrent requests

#3

Yeah, I was thinking of that as a backup solution. Problem is that it requires a complete rewrite of the fetching implementation.

I have to determine appropriate block ranges (varying activity), then for each in parallel iterate each cursor until they are depleted, then potentially reversing each segments order and stitch them back together.

The current implementation is rather simple (approx. pseudo code)

1. nElements in block range (gets first page implicitly)
2. taskFactories = (new Array(Math.ceil(nElements/pageSize)).fill(0).map((_,index) => requestFactory(offset=(index+1)*pageSize))
3. await Promise.all(taskFactories.map(t => t()))

ps: Noticed that the data field seems to fixed for getContractEvents (was my last issue in my most recent thread) So maybe I can use offset after all.

Thanks for your fast reply as usual!

#4

We are deprecating the usage of offset in general because it has a considerable load on our backend now.

#5

Can you keep it and just charge users more CU? I can see some business upside for fast synchronization options.

#6

We will have to find other solution to get historical data. In the past you could use the sync event functionality with a server. For now we don’t have a solution to get fast historical data for events.

2 Likes
#7