that output is for any web3api call
anyway, you can also call web3api directly as a http request, without using the SDK, and by using the API key, in that case you donāt even need a server
Ok, thatās what we need (weāre using direct http requests anyway). Right now, we appear to be sending the āApplication ID:ā in the X-API-Key header. What should we change it to (thereās lots of similar keys)? The MasterKey or the CLI API key/secret?
a different key, the web3api key
an example on how to do it from cloud code:
Moralis.Cloud.define("test_web3api", async (request) => {
return Moralis.Cloud.httpRequest({
url: 'https://deep-index.moralis.io/api/v2/web3/version',
headers: {
'accept': 'application/json',
'X-API-Key': 'WEB3API_KEY_HERE'
}
}).then(function(httpResponse) {
return httpResponse;
}, function(httpResponse) {
logger.info('Request failed with response code ' + httpResponse.status);
});
})
Hrm. That appears to be the key weāve been using, not the app ID. So the limits are coming from some other cause?
you can look in the response header to see, there are two types of limits
Hereās a response header. Itās incredibly hard to figure out whatās needed to change to make this work properly. Iām going to echo the sentiment here. These limits are badly-defined, badly reported, and therefore impossible to understand
{
"status":429,
"headers":{
"date":"Fri, 04 Mar 2022 21:17:12 GMT",
"content-type":"application/json; charset=utf-8",
"content-length":"34",
"connection":"close",
"x-powered-by":"Express",
"access-control-allow-origin":"*",
"x-rate-limit-remaining-ttl":"49",
"x-rate-limit-remaining-ip-ttl":"56",
"x-rate-limit-used":"429",
"x-rate-limit-ip-used":"425",
"x-rate-limit-limit":"5000",
"x-rate-limit-throttle-remaining-ttl":"1",
"x-rate-limit-throttle-remaining-ip-ttl":"1",
"x-rate-limit-throttle-used":"130",
"x-rate-limit-throttle-ip-used":"130",
"x-rate-limit-throttle-limit":"125",
"x-rate-limit-remaining":"49",
"x-rate-limit-throttle-remaining":"1",
"etag":"W/\"22-A17CvmppE17nAhS6Eu3AUBz7zmU\"",
"vary":"Accept-Encoding",
"cf-cache-status":"DYNAMIC",
"expect-ct":"max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"",
"server":"cloudflare",
"cf-ray":"6e6da32668508cbf-EWR"
},
"text":"{\"message\":\"Rate limit exceeded.\"}",
"data":{
"message":"Rate limit exceeded."
}
}
it looks like too many requests were done in a second and that throttle limit comes in
now it can depend on what type of requests you make, if you try to resync metadata, you will be able to do only 2 requests per second maybe
One idea is that you build in a cue in the SDK to automatically handle these headers and adapt OR make a far easier system. I already spent 6-8 hours trying to adapt to all your limits, and still donāt feel confident that I have a good solution. Feels like time wasted.
1 Like
Explain the throttle limit, please, and any ways to prevent it applying to the cloud code on the server. Should we be using the Moralis SDK rather than http calls? Should we be doing something completely different?
Using http calls is fine. Based on what requests you make could be a different reason for getting that throttle limit that means that too many requests were done in a second. Usually adding a delay between requests removes that error.
Thatāsā¦ not helpful. Iām not sure you understand how these systems need to work. Weāre not talking about a single client with a single set of calls. Weāre in a world with multiple clients, multiple servers, with spiky loads. Thereās no such thing as āput a delay inā. Or, rather, there could be, but it should be your dev team doing those things so your system is fit for purpose.
We can do our best to deduplicate requests (but even thatās a hard problem with the upgraded server, as theyāre multi-instance).
But that still doesnāt help the basic problem that the limits are set up badly, not communicated well, and not configurable for the uses people need. And the only way to figure all this out is spending a ton of engineering time and doing detailed trial-and-error.
You can also increase the limits with a custom plan if you want. The delay you can add it also when you get a rate limit error and try same request after delay.
+1. Moralis team, please be more clear about limits: The pricing page says the āProā package supports 3500 requests per minuteā¦
ā¦But in practice, it is more like 100 per minute. Try to do more and you run into āthrottleā or some other undocumented limit.
My program analyzes <100 wallets using Node.js + Moralis SDK. I hit the rate limit error after about 20 wallets. I waited ~10 min to start over, but the rate limit is still there for even 1 request.
error {
code: 141,
error: 'This Moralis Server is rate-limited because of the plan restrictions. See the details about the current rate and throttle limits: {"x-rate-limit-limit":"3500","x-rate-limit-remaining":"51","x-rate-limit-remaining-ttl":"51","x-rate-limit-throttle-limit":"88","x-rate-limit-throttle-remaining":"1","x-rate-limit-throttle-remaining-ttl":"1","x-rate-limit-throttle-used":"108","x-rate-limit-used":"40"}'
}
This error message is incredibly poorly formatted. Is this really the best naming scheme we could come up with, x-rate-limit-throttle-remaining-ttl?
The documentation here covers exactly two of the dozen or so elements in the error message. What we need is a reference that defines all of them with precision. Hereās another customer asking for the same thing.
If Iām reading this correctly, it says Iāve used only 40 requests over the past minute (āx-rate-limit-usedā:ā40ā) so if Iām paying for 3500/min, why on earth am I getting rate-limited?
At this point, I would expect a response from support that says something like āsome requests count doubleā and ācheck your cloud limitsā but these are not the point. The pricing page says 3,500 requests per minute. I would challenge the Moralis team to show us some code that gets anything close to that. Using any request, cloud limit or other parameters that are available to paying customers like us.
it seems like some extra throttle limit is counted for those requests.
it also looks like we have a problem now with the headers that are returned when using web3api with a server, it looks like some fields are missing.
Lots of āseems likeā and ālooks like.ā Is the problem on the Moralis side, or with our code? If on the Moralis side, whatās the ETA for a fix. If on our side, what do we need to do?
Can someone at Moralis address the rest of the questions in the post?
1 Like
Can you ask one question at a time?. It is clear from that response that throttle limit was hit.
Can you answer the question fully please?
Is the problem on the Moralis side, or with our code? If on the Moralis side, whatās the ETA for a fix. If on our side, what do we need to do?
Can we get a definition of all the limit headers?
Can you show how to get to 3500 requests per minute or is that not possible?
The headers that you see there are not complete. The limit should work as expected. I donāt have an ETA.
I donāt understand. Does this mean the problem is on the Moralis side, or with our code? If on our side, what do we need to do?
Can we get a definition of all the limit headers?
Can you show how to get to 3500 requests per minute or is that not possible?
Those are compute units and not requests. Every request has a different number of compute units. And some requests also have throttle limit that is bigger than the compute unit number.
There is a list in documentation with the number of compute units per API endpoint.