Server >> web api rate limits

Technical Issue Reporting
#1

How can I manage the rate limits between the server and the web3 api? We have server-driven functions that call the API. Iā€™d like to give the server significantly more latitude to make API calls than a client (if the server uses up 2000 calls/minute out of the 5000 we have, thatā€™s fine, but I only want a client to be able to make 5 or 10 calls/minute

We have no authenticated users in our system (and we wonā€™t), so if we can find a way to treat the server as an authenticated user, could that help?

#2

what do you mean by server?

you can limit the clients now in the authenticated or unauthenticated form

#3

Cloud code, running on an upgraded server. I canā€™t tell whether the cloud code is treated as authenticated or not, and what the various limits are. I think Iā€™m setting the limits in an ok manner:

  Moralis.settings.setAPIRateLimit({
    anonymous:10, authenticated:4000, windowMs:60000
  })
#4

ok, so you run some code in cloud code that makes requests to web3api, and you get rate limited in that cloud code with 10 requests per minute?

#5

Yep, I think thatā€™s whatā€™s happening.

#6

Thereā€™s a relevant log, which might confirm that:

2022-03-04T20:50:03.463Z - Ran cloud function coreservices_getWeb3ApiToken for user undefined with:
  Input: {}
  Result: {"status":200,"data":{"success":true,"result":"XXXtokenXXX"}}
#7

that output is for any web3api call

anyway, you can also call web3api directly as a http request, without using the SDK, and by using the API key, in that case you donā€™t even need a server

#8

Ok, thatā€™s what we need (weā€™re using direct http requests anyway). Right now, we appear to be sending the ā€œApplication ID:ā€ in the X-API-Key header. What should we change it to (thereā€™s lots of similar keys)? The MasterKey or the CLI API key/secret?

#9

a different key, the web3api key

an example on how to do it from cloud code:

Moralis.Cloud.define("test_web3api", async (request) => {
  
  return Moralis.Cloud.httpRequest({
   url: 'https://deep-index.moralis.io/api/v2/web3/version',
   headers: {
    'accept': 'application/json',
    'X-API-Key': 'WEB3API_KEY_HERE'
   }
 }).then(function(httpResponse) {
    return httpResponse;
 }, function(httpResponse) {
   logger.info('Request failed with response code ' + httpResponse.status);
 });

})
#10

Hrm. That appears to be the key weā€™ve been using, not the app ID. So the limits are coming from some other cause?

#11

you can look in the response header to see, there are two types of limits

#12

Hereā€™s a response header. Itā€™s incredibly hard to figure out whatā€™s needed to change to make this work properly. Iā€™m going to echo the sentiment here. These limits are badly-defined, badly reported, and therefore impossible to understand

{
   "status":429,
   "headers":{
      "date":"Fri, 04 Mar 2022 21:17:12 GMT",
      "content-type":"application/json; charset=utf-8",
      "content-length":"34",
      "connection":"close",
      "x-powered-by":"Express",
      "access-control-allow-origin":"*",
      "x-rate-limit-remaining-ttl":"49",
      "x-rate-limit-remaining-ip-ttl":"56",
      "x-rate-limit-used":"429",
      "x-rate-limit-ip-used":"425",
      "x-rate-limit-limit":"5000",
      "x-rate-limit-throttle-remaining-ttl":"1",
      "x-rate-limit-throttle-remaining-ip-ttl":"1",
      "x-rate-limit-throttle-used":"130",
      "x-rate-limit-throttle-ip-used":"130",
      "x-rate-limit-throttle-limit":"125",
      "x-rate-limit-remaining":"49",
      "x-rate-limit-throttle-remaining":"1",
      "etag":"W/\"22-A17CvmppE17nAhS6Eu3AUBz7zmU\"",
      "vary":"Accept-Encoding",
      "cf-cache-status":"DYNAMIC",
      "expect-ct":"max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"",
      "server":"cloudflare",
      "cf-ray":"6e6da32668508cbf-EWR"
   },
   "text":"{\"message\":\"Rate limit exceeded.\"}",
   "data":{
      "message":"Rate limit exceeded."
   }
}
#13

it looks like too many requests were done in a second and that throttle limit comes in

now it can depend on what type of requests you make, if you try to resync metadata, you will be able to do only 2 requests per second maybe

#14

One idea is that you build in a cue in the SDK to automatically handle these headers and adapt OR make a far easier system. I already spent 6-8 hours trying to adapt to all your limits, and still donā€™t feel confident that I have a good solution. Feels like time wasted.

1 Like
#15

Explain the throttle limit, please, and any ways to prevent it applying to the cloud code on the server. Should we be using the Moralis SDK rather than http calls? Should we be doing something completely different?

#16

Using http calls is fine. Based on what requests you make could be a different reason for getting that throttle limit that means that too many requests were done in a second. Usually adding a delay between requests removes that error.

#17

Thatā€™sā€¦ not helpful. Iā€™m not sure you understand how these systems need to work. Weā€™re not talking about a single client with a single set of calls. Weā€™re in a world with multiple clients, multiple servers, with spiky loads. Thereā€™s no such thing as ā€œput a delay inā€. Or, rather, there could be, but it should be your dev team doing those things so your system is fit for purpose.

We can do our best to deduplicate requests (but even thatā€™s a hard problem with the upgraded server, as theyā€™re multi-instance).

But that still doesnā€™t help the basic problem that the limits are set up badly, not communicated well, and not configurable for the uses people need. And the only way to figure all this out is spending a ton of engineering time and doing detailed trial-and-error.

#18

You can also increase the limits with a custom plan if you want. The delay you can add it also when you get a rate limit error and try same request after delay.

#19

+1. Moralis team, please be more clear about limits: The pricing page says the ā€œProā€ package supports 3500 requests per minuteā€¦

image
image1213Ɨ719 60.8 KB

ā€¦But in practice, it is more like 100 per minute. Try to do more and you run into ā€œthrottleā€ or some other undocumented limit.

My program analyzes <100 wallets using Node.js + Moralis SDK. I hit the rate limit error after about 20 wallets. I waited ~10 min to start over, but the rate limit is still there for even 1 request.

error {
  code: 141,
  error: 'This Moralis Server is rate-limited because of the plan restrictions. See the details about the current rate and throttle limits: {"x-rate-limit-limit":"3500","x-rate-limit-remaining":"51","x-rate-limit-remaining-ttl":"51","x-rate-limit-throttle-limit":"88","x-rate-limit-throttle-remaining":"1","x-rate-limit-throttle-remaining-ttl":"1","x-rate-limit-throttle-used":"108","x-rate-limit-used":"40"}'
}

This error message is incredibly poorly formatted. Is this really the best naming scheme we could come up with, x-rate-limit-throttle-remaining-ttl?

The documentation here covers exactly two of the dozen or so elements in the error message. What we need is a reference that defines all of them with precision. Hereā€™s another customer asking for the same thing.

If Iā€™m reading this correctly, it says Iā€™ve used only 40 requests over the past minute (ā€œx-rate-limit-usedā€:ā€œ40ā€) so if Iā€™m paying for 3500/min, why on earth am I getting rate-limited?

At this point, I would expect a response from support that says something like ā€œsome requests count doubleā€ and ā€œcheck your cloud limitsā€ but these are not the point. The pricing page says 3,500 requests per minute. I would challenge the Moralis team to show us some code that gets anything close to that. Using any request, cloud limit or other parameters that are available to paying customers like us.

#20

it seems like some extra throttle limit is counted for those requests.

it also looks like we have a problem now with the headers that are returned when using web3api with a server, it looks like some fields are missing.