Scrape NFT token_uri data into Moralis db

I’ve seen this mentioned somewhere on Discord and not sure if it’s already in your backlog, but I needed it so made my own version.

What does it do?
It stores NFT metadata in the database!
I need NFT metadata that is contained in the token_uri json file in my database, so I can use it in queries for filtering and sorting, for example sorting on Name or even searching in the Description.
At the same time it will speed up my app because I get the data from the query and don’t have to wait client side to load the token_uri data after the query.

My approach
I prefer this data in its own Class/table and call it ‘EthNFTMetadata’. I’d like to keep EthNFTOwners as clean as possible since this is auto generated by Moralis. In my query I can easily reference and sort to this data using a ‘lookup’ in an aggregate. I can also easily create BscNFTMetadata and the likes.
I’ve made 2 things:

  1. a Job to check all EthNFTOwners items and scrape the token_uri data into the the EthNFTMetadata Class. You need this if you already have data in EthNFTOwners and you’re not starting from scratch. Or maybe when you migrate to a new fresh server and EthNFTOwners data is repopulated from blockchain. Don’t forget to first create the Class ‘EthNFTMetadata’ in the Moralis db via the Dashboard. This Job code can be added to you Cloud Functions and you can add and run a new Job under Jobs > Schedule a job. You just need to run it once if you also use the BeforeSave hook mentioned in Step 2.
// add token_uri metadata to database for caching and filtering
Moralis.Cloud.job('addMetadata', async (request) =>  {
  logger.info('Running addMetadata job')
  const query = new Moralis.Query('EthNFTOwners')
  query.equalTo('token_address', request.params.input.token_address) // only scrape own tokens
  query.doesNotExist('isScraped') // only check those that have not been scraped before
  const items = await query.find({useMasterKey:true})
  for (let i = 0; i < items.length; ++i) {
    const token_uri = items[i].get('token_uri')
    if (token_uri && token_uri.length) {
      // get token_uri data
      const result = await Moralis.Cloud.httpRequest({
        url: token_uri,
        headers: {
          'Content-Type': 'application/json;charset=utf-8'
        }
      })
      // add new EthNFTMetadata item
      const EthNFTMetadata = Moralis.Object.extend('EthNFTMetadata')
      const metadata = new EthNFTMetadata();
      metadata.set('token_address', items[i].get('token_address'))
      metadata.set('token_id', items[i].get('token_id'))
      metadata.set('name', result.data.name)
      if (result.data.description) metadata.set('description', result.data.description)
      if (result.data.image) metadata.set('image', result.data.image)
      if (result.data.external_url) metadata.set('external_url', result.data.external_url)
      if (result.data.animation_url) metadata.set('animation_url', result.data.animation_url)
      if (result.data.background_color) metadata.set('background_color', result.data.background_color)
      if (result.data.traits) metadata.set('traits', result.data.traits)
      if (result.data.unlockable) metadata.set('unlockable', result.data.unlockable)
      await metadata.save(null, {useMasterKey:true})
      // flag item was scraped
      items[i].set('isScraped', true)
      await items[i].save(null, {useMasterKey:true})
    }
  }
})
  1. a beforeSave hook to scrape and save token_uri data when a new NFT has been minted:
// set Metadata after NFT is created
Moralis.Cloud.beforeSave('EthNFTOwners', async (request) => {
  // add new Metadata item if not scraped already
  if (!request.object.get('isScraped')) {
    const token_uri = request.object.get('token_uri')
    if (token_uri && token_uri.length) {
      // get token_uri data
      const result = await Moralis.Cloud.httpRequest({
        url: token_uri,
        headers: {
          'Content-Type': 'application/json;charset=utf-8'
        }
      })
      if (result && result.data) {
        const EthNFTMetadata = Moralis.Object.extend('EthNFTMetadata')
        const metadata = new EthNFTMetadata();
        metadata.set('token_address', request.object.get('token_address'))
        metadata.set('token_id', request.object.get('token_id'))
        metadata.set('name', result.data.name)
        if (result.data.description) metadata.set('description', result.data.description)
        if (result.data.image) metadata.set('image', result.data.image)
        if (result.data.external_url) metadata.set('external_url', result.data.external_url)
        if (result.data.animation_url) metadata.set('animation_url', result.data.animation_url)
        if (result.data.background_color) metadata.set('background_color', result.data.background_color)
        if (result.data.traits) metadata.set('traits', result.data.traits)
        if (result.data.unlockable) metadata.set('unlockable', result.data.unlockable)
        await metadata.save(null, {useMasterKey:true})
        // flag item was scraped
        request.object.set('isScraped', true)
      }
    }
  }

If you then want to get the NFT with metadata, you can use aggregate and pipeline like so:

const query = new Moralis.Query('EthNFTOwners')
const pipeline = [
    {
      lookup: {
        from: request.params.network+'NFTMetadata',
        let: { token_id: '$token_id', token_address: '$token_address' },
        pipeline: [
          { $match:
            { $expr:
              { $and:
                [
                  { $eq: [ '$token_id',  '$$token_id' ] },
                  { $eq: [ '$token_address', '$$token_address' ] }
                ]
              }
            }
          },
          { $project: { _updated_at: 0, _created_at: 0, ACL: 0, _id: 0 } }
        ],
        as: 'metadata'
      }
    },
    {
      project: {
        token_id: 1,
        token_address: 1,
        token_uri: 1,
        owner_of: 1,
        metadata: { $first: '$metadata' },
      }
    },
    { sort : { metadata.name: 1 } },
]
const queryResults = await query.aggregate(pipeline)

Now you can also sort your query on metadata.name for example!

If you want to see more in depth sorting, for example on askingPrice, see my post at Collation for aggregate([<pipeline>]) not supported by Moralis?

2 Likes

Nice work matiyin. FYI it’s possible to have functions outside the Moralis.Cloud calls. So you could extract an addEthNFTMetadata() function and use it in both places.

Support for BSC and Polygon coming very soon for the Web3API NFT endpoints. The indexing infrastructure is being set up now and is almost ready for release. Then you’ll be able to use those endpoints to populate your own DB. An index job is queued the first time a contract address is requested. Requests will return the partial results until it has been fully indexed.

/nft/{address}
/nft/{address}/{token_id}

Yes you are right about using 1 function!

Great news about having access more end points! I don’t use them yet but will consider in next refactoring stage after first live trail. I’m quite happy with what Moralis already could do from the day I started to use it months ago. Timely showstopper bug fixing and amazing support and growing community. I don’t know how anyone could keep up with all those channels :muscle: :heart_eyes:

3 Likes