NFT Endpoints Temporary Offset Rate Limit

Calling:
moralis.Web3API.token.getAllTokenIds
original path: /nft/{address}
every 1 second and it failed after 51-53 requests (that is almost a minute).
I tried to have a 2-3 sec timeout between the calls but it still eventually failed with a rate limit error.

According to documentation, it’s a 5 weight call that means I should be able to do around 11\sec.

an example of the code running:

const collAddressSlug = '0xf1268733c6fb05ef6be9cf23d24436dcd6e0b35e',
  limit = 50;

let offset = 0;

while (true) {
    const nfts = (await moralis.Web3API.token.getAllTokenIds({ address: collAddress, limit, offset })).result;

   // do something....

    if (nfts.length < limit) {
      break;
    }

  offset += limit;
  await new Promise(f => setTimeout(f, 3000));
}

For pagination there is a different rate limit formula depending on the page number. Higher page numbers have a bigger impact

thanks a lot for the fast reply

  1. what is the rate limit?
  2. it’s seems to be a bit too low. a few minutes to fetch 10K assets.

10k assets should work with a delay of 1 second between requests

It doesn’t, please checkout the code on the original post

You can use a limit of 500 instead of 50

We have just released the possibility to set rate limits for web3-api calls.
See https://docs.moralis.io/moralis-server/web3-sdk/rate-limit on how to configure this.

By default it is set to 50 requests per minute for authenticated users.

I want to do as much as call as I can.
limiting myself is actually the opposite.
I must mention that this is a backend call (node.js).

These rate-limit checks are handled on the moralis server, so it’s the same for node.js
If you don’t want to limit yourself, then you can increase authenticated and/or decrease windowMs

Moralis.settings.setAPIRateLimit({
  anonymous:30, authenticated:500, windowMs:1000
})

Will give you 500 requests per second for example.

I can’t find “settings” inside Moralis package, is it out yet? any specific version?
(I see the documentation was added few hours ago)

  • I’m using version 0.0.157 now

You may need to also update your Moralis server instance. I think that code is specific for cloud code and not for front end.

I mentioned using Web3API.token.getAllTokenIds specifically.

so what’s next?

did you try with a limit of 500 and with a delay of 1 second between requests?

yes, but why do I even need a 1 sec delay?
I need a clear understanding of the mechanism and what the limitations are?

Thanks

it looks like currently the weight is multiplied by offset / 500

I think that we plan on doing some improvements soon on pagination and we may lower that weight after that

At the moment we temporary restriction on some endpoints (like NFT endpoints) pagination while we scale the systems:

The weight of the NFT endpoint that uses offset is calculated like this (offset/500)*5

if the result of the above calculation plus the weight of other requests you made the same second is above 60 - wait 1 second

if more requests are sent before 1 second i over - wait 5 seconds

these are temporary restrictions we had to put in place as we scale our systems and will be removed in the coming week approximately

1 Like

Is there any ETA on when this weighting will be removed?

some NFT endpoints now have cursor based pagination (from changelog in discord):

  NFT transfer endpoints are automatically order by timestamp descending.
  Cursor based pagination is now supported for those endpoints
       - getNFTTransfersByBlock /block/{block_number_or_hash}/nft/transfers
       - getNftTransfersFromToBlock /nft/transfers
       - getNFTTransfers /{address}/nft/transfers
       - getContractNFTTransfers /nft/{address}/transfers
       - getWalletTokenIdTransfers /nft/{address}/{token_id}/transfers

Is there any ETA on when the getAllTokenIds endpoint will have the offset weighting for request throttling removed?

I don’t have an ETA now for that