Performance related to sync BscTransactions/BscTokenTranfers

Technical Issue Reporting
#1

Hi,

First of all, I want to thank the Morales team for creating such a good framework which makes our DEV’s life so much easier for creating dApp.

We are launching our mainnet dApp on 6/11. (the first small event already launched yesterday 1/11). We are facing the critical problem:

  • Our instance is the big one: 499usd/month
  • We have 3k users which logged in during 3h of event
  • I realise that everytime a new user created, Morais sync all of BscTransactions/BscTokenTranfers of that user so after only 3h of event, we have about 300k BscTransactions and 110k BscTokenTransfer (which are not related to our token at all because we did not sell our token yet)
    => and when there are many users created at the same time, these process block our server, the server become inaccessible and our dApp is down.

So my suggestion is:

  • Filter BscTokenTransfers to sync only transfers related to our contract (should be able to provide it by an input)
  • Being able to filter only needed BscTransactions to sync
  • Being able to disable sync BscTransactions when unneeded

Because with the current state, our dApp is blocked and not usable. It’s really critical for us. Please help us on this point. Many thanks in advance.
Yo.

#2

Disabling historical sync is an option that you could use now?

#3

You could sync your contract transactions and nft owners separately as a work around now using web3api calls and inserting that info in your database and completely disabling historical sync

#4

Hi, yea it looks like you are syncing unncesessary data
Im not sure why you need all that data in the DB
500k events and also in real time need a big machine

Historical sync feature of the server is there if you need to sync everything about the user - which you don’t need. We will add more filtering options in the future, there is no filtering options now.

SOLUTION:

you can get data related to your token contract only using the Web3 API
https://docs.moralis.io/moralis-server/web3-sdk/token

There is no need to sync into database

the way to solve this is to disable historic sync (as it’s not your use case - historic sync is only needed if you want all transactions from the users)

and use only API to get relevant data

Let us know if this works!

#5

Also - another solution is doing a sync event only for your smart contract and disabling historical user sync https://docs.moralis.io/moralis-server/automatic-transaction-sync/historical-transactions#history-and-activity-related-only-to-your-contract

We are here to help whatever solution you choose - and soon we will add support for filters :star_struck:

#6

Thanks for your support. I will check these solutions :slight_smile:

#8

this is disabled as well

Screenshot 2021-11-04 at 23.20.18
Screenshot 2021-11-04 at 23.20.181720×708 57.4 KB

#9

This now becomes very critical for us because our launching will be in 2days and if Moralis keep syncing everything like this, the server will be crashed/down again like what happened 3 days ago when there was many user logged in at the same time. Could you please check if the sync process is not disabled or I did something wrong ?

Thanks in advance.

#11

Should be no more historical transactions synced now on your server (we cleared the queue), and any custom events you add should work correctly. Let us know you have other problems.

#12

Hi @cryptokid,
It works now like a charm. Thank so much. You guys are really AMAZING !!!

1 Like
#15

We may have to fix something for this to work, are you sure you want to sync a contract address this way and not using events sync? Or this was only a test to see if it works.

#16

@cryptokid,
Basically, I’d like to sync all the transactions related to our token (using BscTokenTransfers with token_address = our token). Our dApp based on that logic already (using data in BscTokenTransfers to sync with our data). Just like option Sync that we have already disabled, except that option Sync synced not only the transactions related to ours but also all others user transactions which caused the problem of performance.

Our big launching will be in 12h so it’s quite short to change or test something else :frowning: So could you please fix something for this to sync only transactions related to our tokens into BscTokenTransfers ? Thank so so so so much in advance.

P/S: I did not base on contract event to sync because I thought that if an event emitted while the server is restarting, we will lose the event ? but all the ways, the time is too short for us now to change the logic. :frowning:

1 Like
#18

If transactions will start to sync on your server for all the addresses, but without historical transactions, that would work fine for you?
That would mean that you will also not have historical data for your contract address.

#19

what was your solution? we have the same ´problem bro, pls.

#20

only as an update: we discussed on Discord and user sync was disabled on that particular server