Extra token transfer event synced (never gets confirmed)

[barclander]

See transaction https://polygonscan.com/tx/0xffb95c89454b0cdfda2b69c9fc83a2a001497a5d982dafe3283a9f9b2a02c632

I see bad data being synced on server id # https://xc5a0mdc061u.usemoralis.com

There’s two GAME and two GENESIS erc-20 transfer events in that transaction. We sync both contracts’ events on the server (standard sync). In both tables (TransferGameEvent and TransferGenesisEvent), there are two events listed with the correct transaction hash when there should only be one. However, in both places, they show incorrect data, the block hash/number is wrong (bad but not fatal), but the value is also wrong (critical error). The wrong event is not listed as confirmed (which makes sense, because it’s just bad data), but it’s an extra event, so when we track the pending balances, it’s wrong

We need this fixed, as it makes erc-20 balances incorrect until 4 minutes after the transaction completes, which is not acceptable.

[cryptokid]

That unconfirmed transaction looks strange, don’t know yet why is a wrong block_number and wrong value.
For the confirmed transaction everything is right?

[barclander]

Yep, the other events are all correct, it’s just these weird ones that are messed up.

I tested on a fresh server; the historical events were pulled correctly; it’s definitely an error in the “watch” part of the code.

Want to know the really mind-bending thing here? The “incorrect” event shows as being created (and last updated) 1 minute and 14 seconds before the “correct” event is created, and 1 minute 12 seconds before the “correct” event was emitted on chain. It shouldn’t be possible to manage that - the tx hash wasn’t known to the world until the correct event was emitted, over a minute after the incorrect event was last updated.

It looks like the bad object was somehow overwritten in mongodb with a couple of data points (txhash & address) from the good object, without running through the normal “save”process.

But it’s not a persistent error - it didn’t corrupt the actual event storage (because the historical sync later was right).

Still, it’s a pretty critical bug to get fixed.

[cryptokid]

We are going to investigate, thanks for reporting

[cryptokid]

After some investigation it looks like this happened because a re-org on the chain and in the future we will have automatic removal of these transactions.

[barclander]

Automatic removal isn’t going to help if it gets synced into the database at any point - if we’ve already done an operation based on it being saved as that value, we need to be able to undo that operation

Hmm. I guess we could do a scheduled job to undo and delete unconfirmed events older than X. Running that every 5 or 10 minutes is probably fine; these are very low occurrence and low impact events.

We could also do that if you edit “undefined” confirms to “false” confirms; we trigger on that change, delete the object, and unwind its effect on the DB.

So either way, we’ve got a mitigation that can be done. LMK what direction you want to take (just don’t remove them from the DB silently because that destroys any ability to fix the issue)

[barclander]

Update: I’ve put in cron jobs to undo and invalidate unconfirmed events. We’d love to be able to trigger on a switch from “unconfirmed” to “false”, as that would get data corrected ~10 mins faster (and be more reliable).
The cron job doesn’t delete the erroneous events, it just tags them with a data processing ID that’s higher than can ever be used; so we can still investigate if you need to.