504's and and cloudflare blocking requests

Technical Issue Reporting
#13

For what url in particular you get errors?

#14

Most recently Web3API.token.getTokenPrice. But it seems fairly random. We make use of a substantial portion of the Web3API calls.

But the bigger issue is that the IP gets blocked. Not sure if it’s permanent, but long enough that we must change it, and of course manually change the IP for Mongo access.

#15

Can you check the http headers for when you get those errors? How many requests do you make?

You shouldn’t get easily blocked for that request unless you make thousands per second

#16

I’ve already cleared the most recent series of errors and logs. Letting it run again overnight and I’ll check it in the morning.

Not making thousands per second, but some bursts between 50-200 per second. We’re making various calls regarding Swap “actors”. Token prices, holder information, etc. – for multiple token swaps at a time.

I’m offloading as much as possible to direct contract calls, but we can’t always rely on a contract to be verified or the source to be accessible, so we rely on Moralis in those instances.

Regardless, it would be nice to know what the explicit limitations were in all circumstances so as to code around it (or drop the feature(s)).

#17

You don’t get rate limited for 200 requests per second?

#18

FYI, this is my second time posting this – from the same test IP. The first reply apparently got blocked due to, well, CF/Moralis blocking traffic. Swapped my IP and now this works.

Yes, I will occasionally get transactions rejected during bursts, but CF doesn’t usually step in and completely block my IP in those situations. But, for some reason, at some point, CF/Moralis decides it doesn’t want to hear from us anymore and simply blocks the IP.

Here is the most recent error output. Not sure if it pertains to the issue, but it happened hours ago while I was asleep:

2022-08-29T06:44:20.375Z - Error: {}
    at fetch (/moralis-server/lib/cloud-code/plugins/helpers/apiUtils.js:218:26)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
2022-08-29T06:44:20.368Z - Failed running cloud function getTokenBalances for user undefined with:
  Input: {"chain":"bsc","address":"0xD4825ad5B1A863587e7116f3DE1387C1238e4319","to_block":"20858573"}
  Error: {"message":"{}","code":141}
2022-08-29T04:57:32.391Z - Error: Invalid function: "getPluginSpecs"
    at handleCloudFunction (/moralis-server/lib/Routers/FunctionsRouter.js:201:13)
    at /moralis-server/lib/PromiseRouter.js:85:20
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

For what it’s worth, Get all logs in your terminal: Learn about Moralis CLI is a dead link and needs to be fixed.

#19

you are using a Moralis Server to make web3api requests from a backend?
asking because that looks like server log

in that case what IP was blocked, your local IP, server IP?

#20

It’s a node application. Yes, just using the existing Web3API collection (rather than axios/fetch with endpoints on the backend). Is the EVM object available in V1? What do you recommend?

There are a few devices (hardware and docker) behind the same VPN’ed firewall so they’re all sharing the same external IP. But only one VM is interacting with Moralis. It’s blocking that shared VPN IP

#21

I recommend to use web3api directly without a Moralis Server, either using moralis v1 sdk with moralisSecret parameter, or REST API, or moralis v2 SDK

#22

But what about the various API-limited calls? Don’t I need server credentials to increase limits?

Is the REST API not “limited”?

This is basically what I’m using:

const Moralis = require('moralis-v1/node');
const options = {
  serverUrl: dAppServerUrl,
  appId: appId,
  masterKey: masterKey,
};

await Moralis.start(options);

I’m using that object to then make the majority of calls. What’s the problem with that specifically? :thinking:

#23

you should use only moralisSecret for those options if you only use web3api

you don’t need to increase those limits, those limits can not be increased more than the limits that you have for your account, increasing them more will make it hit the account rate limit

#24

These are the relevant calls we make to the Moralis object from the app:

...Web3API.token.getTokenMetadata

...Web3API.token.getTokenPrice

...Web3API.native.getTransaction

...Web3API.account.getTokenBalances

...Web3API.native.getLogsByAddress

...Plugins.oneInch.getSupportedTokens

...Web3API.defi.getPairAddress

...Web3API.defi.getPairReserves

...Cloud.run

You mind providing a specific example of one that you’d swap out for a different SDK or call?

#25

it looks like you don’t use only web3api, there is also 1Inch and running cloud functions

#26

The 1Inch call is a raaaaaare call. Like weekly. The only cloud functions involve boot-up coreservices_addEventSync just to ensure that sync actions are running for retrieval later via Mongo.

The native/token/account calls are what generate the traffic.

#27

I mean that you can not completely stop using the server now, you are still using it for other things besides web3api

for web3api you should get a specific error message when you hit the rate limit

#28

I think we might be conflating two issues:

  1. Moralis rate-limit

  2. CloudFlare IP banning

My primary issue isn’t rate-limiting . . . I expect this to happen. The primary issue is that CF randomly decides to ban the IP, and I can no longer access Moralis things. I have to change my IP to regain access then everything works fine again.

#29

If you stop making requests when you hit the rate limit then you shouldn’t get any ban. I think that you will get a ban if you make too many requests, like thousands in a second.

#30

So, basically (pricing aside), if I’m looking for “wide open” data access, Moralis is not what I need?

Do direct Mongo calls count against the IP?

#31

Direct mongo calls don’t count towards rate limit. The rate limit doesn’t seem to be the issue now, more like it is ignored. You can get a higher rate limit with a better plan if needed.
But calls to the server could trigger cloudflare protections as the server is behind cloudflare.

#32

Yeah, that’s my concern. Regardless of tier, I might still get CF’ed. Is it possible to get an IP “whitelisted” all the way through? :thinking: