Bringing back EthNFTOwners sync on a Nitro server

I started a new fresh server instance on v0.0.369. I’ve minted NFT’s but the EthNFTOwners table is not created. The mints tx do appear in EthTransactions, exist on chain and also appear in the web3 api endpoint https://deep-index.moralis.io/api/v2/<wallet_address>/nft?chain=rinkeby

I’ve run
await Parse.Cloud.run('watchEthAddress', {address: '<wallet_address>', 'chainId':'0x4', 'sync_historical': true}, { useMasterKey: true }) through js console, no avail, even after hours.

What happens: it creates a new table structure, instead of watchedEthAddress still on my other v0.0.359 server, there’s now _AddressSyncStatus.

v0.0.359 server:

v0.0.369 server:

It’s unclear how this works, but I believe it’s related to the watch address historical sync job. No idea why it only mentions ERC1155 and not ERC721 in the field names, but it’s always been like that.

I’m unsure how to debug further. NFTOwners table not population has been on ongoing topic. I’ve had it myself a few times before, sometimes a moralis issue, sometimes had to run the watchEthAddress manually. But since table names changed, might be a new bug?

Also, there’s no tokens_last_error field anymore in _AddressSyncStatus so I can’t see any errors.

For example, on the old server I once got following error, indicating an issue with fetching from IPFS because of an invalid token_uri url on one of the found NFT’s:

{"status":400,"headers":{"date":"Tue, 15 Feb 2022 11:27:28 GMT","content-type":"text/html; charset=utf-8","transfer-encoding":"chunked","connection":"close","cf-cache-status":"DYNAMIC","expect-ct":"max-age=604800, report-uri=\"https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct\"","server":"cloudflare","cf-ray":"6dde2febea974bfa-AMS"},"buffer":{"type":"Buffer","data":[xxx]},"text":"<!DOCTYPE html>\n<html>\n\t<head>\n\t\t<meta charset=\"utf-8\">\n\t\t<meta http-equiv=\"refresh\" content=\"10;url=/ipfs/QmSVRSs68TBPncnQp8UPXGmhihvnsGEgjiiDu2xNrKd7ke\" />\n\t\t<link rel=\"canonical\" href=\"/ipfs/QmSVRSs68TBPncnQp8UPXGmhihvnsGEgjiiDu2xNrKd7ke\" />\n\t</head>\n\t<body>\n\t\t<pre>invalid path: &#34;/ipfs/ipfs/QmSVRSs68TBPncnQp8UPXGmhihvnsGEgjiiDu2xNrKd7ke&#34; should be &#34;/ipfs/QmSVRSs68TBPncnQp8UPXGmhihvnsGEgjiiDu2xNrKd7ke&#34;</pre><pre>(if a redirect does not happen in 10 seconds, use \"/ipfs/QmSVRSs68TBPncnQp8UPXGmhihvnsGEgjiiDu2xNrKd7ke\" instead)</pre>\n\t</body>\n</html>"}

If need I can provide the server instance through Discord.

Thanks a bunch.

Hi,

Any new created server will be a nitro server now, and nitro servers in particular don’t have EthNFTOwners any more.

There is some info in this forum post about what changed in a nitro server:

@cryptokid @ivan wow I’m quite speechless…

What does this mean for my app that’s finally ready to be used in production with my first client?
I’ve worked on your infrastructure for over a year and you just decide to remove some core functionality without backwards compatibility?

What happens when I update my ‘old’ 0.0.359 server? It will convert to ‘Nitro’ and stop syncing NFTOwners and NFTOwnersPending?

Can I still get a new server that is not Nitro or do I need to rewrite my app and add those NFTOwners tables myself now on mint events?

I did not want to rely on the web3 api too much. It’s an overkill for my app, that’s just managing NFTs from particular contracts. I have no need for historical info normally, only in the event I need to migrate server and repopulate user content. I like the idea of storing the NFT data in my own database. It’s fast and cached, I can add my own data and no need to query the web3 api each time to get user content that doesn’t change that much. The web3 api can be down, it can have bugs, it can cost me usage each time I call it, while now I just call the database.

I understand you’re focusing more and more on ‘ease of use’, a toolkit based on React with ‘plug and play’ components to develop simple apps fast. But I’m not that user and feel a bit ‘rugged’.

Yes of course I can adapt, I can rewrite, but wow this kinda kills my confidence in building on Moralis. Maybe I’m not your target group and only using a sub set of your features. I’m wondering how I can build a business on something that just throws in ‘breaking changes’ like that…

any new created server will be a nitro server

when you update the server it will not be migrated automatically to a nitro server, you will have to install coreservices plugin on that server to migrate it to the nitro version

Maybe you can use EthNFTTransfers table to get the data that you need.
Also, maybe using an event sync for the mint/transfer event would help.

For others that run into the same ‘breaking’ issue and want the NFTOwners table back, here’s what I did:

Use EthNFTTransfers table to sync mints and transfers in NFTOwners table
Run a beforeSave hook in cloud functions to catch NFT transactions from connected wallets.
Note: I have a marketplace with several Collections (ERC721 contracts) so I first check if the transfer is from a ‘white listed’ nft collection in my system.

// set <chain>NFTOwners table to track minted and transferred NFTs
Moralis.Cloud.beforeSave('EthNFTTransfers', async (request) => {
  logger.info('process EthNFTTransfers')
  logger.info(JSON.stringify(request.object))
  // process transfer if tx confirmed and not a Burn
  if (request.object.get('confirmed') && request.object.get('to_address') != '0x0000000000000000000000000000000000000000') {
    const network = 'Eth'
    const chain = 'rinkeby'
    const chainId = '0x4'
    const apiKey = '<your_moralis_web3_api_key>'
    // check if token is part of app Collections
    const getCollections = new Moralis.Query('Collection')
    getCollections.equalTo('network', chainId)
    getCollections.equalTo('isDeployed', true)
    getCollections.ascending('menu_order')
    const collections = await getCollections.find({useMasterKey:true})
    if (collections) {
      const collection = collections.filter(collection => collection.token_contract === request.object.get('token_address'))
      if (collection) {
        // add minted NFT to NFTOwners table
        if (request.object.get('from_address') === '0x0000000000000000000000000000000000000000') {
          const NFTOwners = Moralis.Object.extend(network+'NFTOwners')
          const newNFT = new NFTOwners()
          // fetch token details from moralis web3 api
          const options = {
            token_address: request.object.get('token_address'),
            token_id: request.object.get('token_id'),
            chain: chain,
          }
          Moralis.Cloud.httpRequest({
            url: `https://deep-index.moralis.io/api/v2/nft/${options.token_address}/${options.token_id}`,
            params: {chain: options.chain},
            headers: {
              accept: 'application/json',
              "X-API-Key": apiKey,
            },
          })
          .then((response) => {
            logger.info(JSON.stringify(response.data))
            if (response.data) {
              // save NFT data to NFTOwners
              newNFT.set('token_id', response.data.token_id)
              newNFT.set('token_address', response.data.token_address)
              newNFT.set('owner_of', response.data.owner_of)
              newNFT.set('token_uri', response.data.token_uri)
              newNFT.set('name', response.data.name)
              newNFT.set('symbol', response.data.symbol)
              newNFT.set('block_number', response.data.block_number_minted)
              newNFT.set('contract_type', response.data.contract_type)
              newNFT.save(null, {useMasterKey:true})
            // }
          }, function(error) {
            logger.info('error getting NFT data from web3 api')
            logger.info(error)
          })
        } else {
          // change owner_of if NFT was transferred to other wallet
          const query = new Moralis.Query(network+'NFTOwners')
          query.equalTo('token_id', request.object.get('token_id'))
          query.equalTo('token_address', request.object.get('token_address'))
          const NFTOwnerItem = await query.first({useMasterKey:true})
          if (NFTOwnerItem) {
            NFTOwnerItem.set('owner_of', request.object.get('to_address'))
            NFTOwnerItem.save(null, {useMasterKey:true})
            logger.info('transferred to new owner '+request.object.get('to_address'))
          }
        }
      }
    }
  }
})

Add a Cloud Job that you can run to do import existing NFTs as NFTOwners
I use this when I migrate to a new server instance and want to manually import existing NFTs from white listed Collections. Also handy for repairing bugs. I don’t run it on a schedule, but it’s an option if you want an additional backup check to process historical.

// add NFTOwners table from deployed contracts
Moralis.Cloud.job('addNFTOwners', async (request) =>  {
  logger.info('Running addNFTOwners job')
  const getCollections = new Moralis.Query('Collection')
  getCollections.equalTo('network', request.params.input.chainId)
  getCollections.equalTo('isDeployed', true)
  getCollections.ascending('menu_order')
  const collections = await getCollections.find({useMasterKey:true})
  if (collections) {
    collections.forEach(collection => {
      const options = {
        token_address: collection.get('token_contract'),
        chain: request.params.input.chain,
      }
      Moralis.Cloud.httpRequest({
        url: `https://deep-index.moralis.io/api/v2/nft/${options.token_address}/owners`,
        params: {chain: options.chain},
        headers: {
          accept: "application/json",
          "X-API-Key": "<your_moralis_web3_api_key>",
        },
      })
      .then(async (response) => {
        if (response.data.result.length) {
          for (let i = 0; i < response.data.result.length; i++) {
            const nft = response.data.result[i];
            const NFTOwners = Moralis.Object.extend(request.params.input.network+'NFTOwners')
            // check if exists
            const query = new Moralis.Query(request.params.input.network+'NFTOwners')
            query.equalTo('token_id', nft.token_id)
            query.equalTo('token_address', nft.token_address)
            const itemExists = await query.first({ useMasterKey: true })
            // save item to NFTOwners
            if (!itemExists) {
              const item = new NFTOwners();
              item.set('token_id', nft.token_id)
              item.set('token_address', nft.token_address)
              item.set('owner_of', nft.owner_of)
              item.set('token_uri', nft.token_uri)
              item.set('name', nft.name)
              item.set('symbol', nft.symbol)
              item.set('block_number', nft.block_number)
              item.set('contract_type', nft.contract_type)
              item.save(null, {useMasterKey:true})
            }
          }
        }
      }, function(error) {
        logger.info('error getting NFTs in contract from web3 api')
        logger.info(error)
      });
    });
  }
})

Because I use a curated collection, I’d like to have the NFT and meta in my db for speed, caching, reliability and storing custom data for my system. Moreover, it takes less resources from servers.
Alternatively you could run the cloud Job on a schedule, instead of using EthNFTTransfers.

1 Like

@matiyin this is awesome thank you.

Would you happen to have a list of all database columns to go along with this so I can figure out a couple of the lines?

hi jins,

not sure what you mean, the database columns that I use in NFTOwners are the ones mentioned in the object nftData in my code. You don’t need to create any columns or table by hand in the Dashboard, just run the code and maybe see if you want to add your own fields for your system.

Here’s the updated and current cloud code I’m using. Note there’s extra code checking if the token found in the beforeSave hook is part of my app’s Collections (smart contracts). If not I don’t save it, since I only deal with a curated set of contracts. It’s easy to take that part out if you want.

async function beforeSaveNFTTransfers(request, network, chainId) {
  logger.info('process '+network+'NFTTransfers')
  const config = await Moralis.Config.get({useMasterKey: true});
  const apiKey = config.get('web3api_key'); // set your Moralis web3 api key in the Dashboard Config section (using Master Key for security)
  const nftData = {
    objectId: request.object.id || null,
    token_id: request.object.get('token_id'),
    token_address: request.object.get('token_address'),
    owner_of: request.object.get('to_address'),
    token_uri: null,
    name: null,
    symbol: null,
    block_number: request.object.get('block_number') || null,
    contract_type: request.object.get('contract_type'),
    wasBurned: null,
  }
  // check if token is part of app Collections
  const getCollections = new Moralis.Query('Collection')
  getCollections.equalTo('network', chainId)
  const collections = await getCollections.find({useMasterKey:true})
  if (collections) {
    const collection = collections.filter(collection => collection.token_contract === request.object.get('token_address'))
    if (collection) {
      if (request.object.get('confirmed')) {
        // add minted NFT to NFTOwners table
        if (request.object.get('from_address') === '0x0000000000000000000000000000000000000000') {
          // fetch token details from moralis web3 api if tx confirmed
          // this can still fail!
          const options = {
            token_address: request.object.get('token_address'),
            token_id: request.object.get('token_id'),
            chain: chainId,
          }
          const apiResponse = await Moralis.Cloud.httpRequest({
            url: `https://deep-index.moralis.io/api/v2/nft/${options.token_address}/${options.token_id}`,
            params: {chain: options.chain},
            headers: {
              accept: 'application/json',
              "X-API-Key": apiKey,
            },
          })
          try {
            const apiData = await apiResponse.data
            // set data from web3 api if valid
            nftData.owner_of = apiData.owner_of
            nftData.token_uri = apiData.token_uri
            nftData.name = apiData.name
            nftData.symbol = apiData.symbol
            nftData.block_number = Number(apiData.block_number_minted)
          } catch(err) {
            logger.error('error getting NFT data from web3 api')
            logger.error(error)
          }
        } else {
          if (request.object.get('to_address') === '0x0000000000000000000000000000000000000000') {
            // process burn
            nftData.wasBurned = true
            logger.info('NFT was burned')
          } else {        
            //process tranfer
            logger.info(`transferred to new owner ${nftData.owner_of}`)
          }
        }
      }

      // save NFT data
      let NFTitem = {}
      if (request.object.get('confirmed')) {
        // check if exists
        const NFTOwnersQuery = new Moralis.Query(network+'NFTOwners')
        NFTOwnersQuery.equalTo('token_id', nftData.token_id)
        NFTOwnersQuery.equalTo('token_address', nftData.token_address)
        const itemExists = await NFTOwnersQuery.first({ useMasterKey: true })
        if (!itemExists) {
          const NFTOwners = Moralis.Object.extend(network+'NFTOwners')
          NFTitem = new NFTOwners()
        } else {
          NFTitem = itemExists
        }
        // delete NFTOwnersPending
        const queryPending = new Moralis.Query(network+'NFTOwnersPending')
        queryPending.equalTo('token_id', nftData.token_id)
        queryPending.equalTo('token_address', nftData.token_address)
        const pendingItem = await queryPending.first({ useMasterKey: true })
        if (pendingItem) {
          pendingItem.destroy({useMasterKey:true}).then(() => {
            logger.info('Pending item was deleted');
          }, (error) => {
            logger.error(error)
          })
        }

      } else {
        // set new pending object
        // check if exists
        const NFTOwnersPendingQuery = new Moralis.Query(network+'NFTOwnersPending')
        NFTOwnersPendingQuery.equalTo('token_id', nftData.token_id)
        NFTOwnersPendingQuery.equalTo('token_address', nftData.token_address)
        const itemExists = await NFTOwnersPendingQuery.first({ useMasterKey: true })
        if (!itemExists) {
          const NFTOwnersPending = Moralis.Object.extend(network+'NFTOwnersPending')
          NFTitem = new NFTOwnersPending()
        } else {
          NFTitem = itemExists
        }
      }
      // save the data to a new db Object
      if (nftData.token_id) NFTitem.set('token_id', nftData.token_id)
      if (nftData.token_address) NFTitem.set('token_address', nftData.token_address)
      if (nftData.owner_of) NFTitem.set('owner_of', nftData.owner_of)
      if (nftData.token_uri) NFTitem.set('token_uri', nftData.token_uri)
      if (nftData.name) NFTitem.set('name', nftData.name)
      if (nftData.symbol) NFTitem.set('symbol', nftData.symbol)
      if (nftData.block_number) NFTitem.set('block_number', Number(nftData.block_number))
      if (nftData.contract_type) NFTitem.set('contract_type', nftData.contract_type)
      if (nftData.wasBurned) NFTitem.set('wasBurned', nftData.wasBurned)
      NFTitem.save(null, {useMasterKey:true})
      .then((obj) => {
        logger.info(request.object.get('confirmed') ? 'NFTOwner saved:' : 'NFTOwnerPending saved:')
        logger.info(JSON.stringify(obj))
      }, (error) => {
        logger.error(error.message)
      })      
    }
  }
}

// set <chain>NFTOwners table to track minted and transferred NFTs
Moralis.Cloud.beforeSave('EthNFTTransfers', async (request) => {
  beforeSaveNFTTransfers(request, 'Eth', '0x5')
})
Moralis.Cloud.beforeSave('AvaxNFTTransfers', async (request) => {
  beforeSaveNFTTransfers(request, 'Avax', '0xa869')
})

Basically what it does:

  • check for newly created NFT’s (transfer from 0x0000…) that are not yet confirmed (they go to NFTOwnersPending)
  • check for confirmation of an NFT and move it from NFTOwnersPending to NFTOwners
  • check for burned items (transfer to 0x0000…)
  • check for transferred items that already exist and update the owner

Also note that I split up the code into a function so I can easily set multi chain hooks. You don’t need that if you just deal with one chain.

Hope that’s helpful and good luck :mage:

@matiyin Amazing thank you

I’m trying to save my authenticated users’ NFTs from a specific collection into a DB table so this helps a lot.

1 Like