Web3 API servers crashing consistently when calling getBlock

Technical Issue Reporting
#1

I’m trying to do something that (I think) should be pretty basic, but it’s consistently causing my web3 api servers to hang after a few hours. It’s happening on bsc, polygon, avax & fantom.

The servers are brand new, not customised at all except to increase user rate limits. I don’t have anything installed on them. All I’m doing is this:

Moralis.start Moralis.Web3API.native.getBlock

Between 0.5-2x per second. After a while this starts happening:

  • requests start taking a long time (60+ seconds)
  • cpu at 100%
  • start getting 5xx errors
  • dashboard no longer works
  • server stops responding & doesn’t come back (despite no more requests being sent)

I don’t think throwing more cpu at the problem is the answer because clearly something is breaking with the server (it doesn’t come back after I stop sending requests).

Can someone shed some light on this?

Server urls:

https://infck3kckkjd.usemoralis.com:2053/server
https://lfemz6zt3brq.usemoralis.com:2053/server
https://jnzaoozqho6t.usemoralis.com:2053/server
https://ltrz7hqcxudm.usemoralis.com:2053/server
#2

how many times can you call that function successfully until this problem happens?

#3

I’ve seen it crash within 30 mins but it can be up to several hours. So that would be 2000 requests give or take?

#4

ok, how long does it take to crash this server?

https://nim98yhfasdo.usemoralis.com:2053/server
EUhN2vhAwUJn83jzBUNfwaT5fsS1GyGJHjbqadDL

this is one of my servers that is an upgraded version

#5

I guess that if you restart/update the server everything comes back to normal

#6

Thanks I’ll let it run on this server for a while.

#7

Seems to be stable so far. Was it just the version that was upgraded, or does this one have higher cpu specs as well?

#8

This one has higher CPU specs

#9

what is the code that you used?

#10

I’m just calling Moralis.start, then Moralis.Web3API.native.getBlock. For 1-2x blocks per second. That’s it. Any ideas why that might be crashing the server? I don’t think it’s a memory leak since mem usage doesn’t go to 100%. But the cpu stays pinned at 100% even after I stop sending requests.

#11

I’m noticing the request time has increased on your server to 45-90 seconds. It was previously 5-10s for fantom & avax. Maybe it’s starting to die as well.

#12

Yes, that server is not working well now, it has some high CPU usage

#13

Can you also try to do a test where you call web3api from a could function using a http request instead of using web3api from sdk?

#14

Ok let me figure out how to do that

#15

Ok so same behaviour when using cloud functions. It takes a bit longer to cause the server to go 100% cpu and stop responding (like 4-8 hrs) but it still happens. This time some of the servers would still respond for a brief minute, but when I start running getBlock on them they almost immediately go unresponsive. Something with the server is obviously not cleaning up after itself.

I think the servers need a dev to look at them when they’re in this state to see what’s consuming the cpu.

#16

How did you call the api from the cloud function?

Did you use the api key with a http request or the web3api call available?

#17

I’m calling it with a http request using the api key

#18

Ok, strange. Now, can you try with a different http request, that only reads the data from an url to see if it replicates?

#19

Ok I’m calling a cloud function via http that only returns the param it was sent. It returns quickly in about 200 ms, cpu is at 10% and so far no degradation.

#21

What’s the next step?