Copy / clone database tables to other server

Moralis General
#1

I’m migrating a server. What’s the recommended way to copy over some tables to a new server, preferable maintaining collection properties and indexes?
Is it possible with tools like https://docs.mongodb.com/database-tools/mongodump or only by writing an export/import cloud function?

#2

In the future you may have direct access to mongo db and you could do that dump.
For now you can do that import/export with a cloud function.

#3

Has anyone worked on a generic function to do this export / import? Seems like it could be built in a general purpose way.

#4

I created a simple cloud function to import/export from one server to another:

// import data from other server
Moralis.Cloud.job('importData', async (request) =>  {
  logger.info('importData')
  const objectClass = request.params.input.objectClass
  const config = await Moralis.Config.get({useMasterKey: true});
  const ApplicationId = config.get('cms_app_id');
  const ServerUrl = config.get('cms_server_url');
  const url = `${ServerUrl}/functions/exportData`
  const result = await Moralis.Cloud.httpRequest({
    url: url,
    params: {
      _ApplicationId: ApplicationId,
      objectClass: objectClass,
    }
  })
  if (result.data && result.data.result) {
    for (let i = 0; i < result.data.result.length; ++i) {
      const objClass = Moralis.Object.extend(objectClass)
      const obj = new objClass()
      // strip things we dont want
      delete result.data.result[i].createdAt
      delete result.data.result[i].updatedAt
      delete result.data.result[i].objectId
      delete result.data.result[i].__type
      delete result.data.result[i].className
      // set each field
      for (const key in result.data.result[i]) {
          obj.set(key,result.data.result[i][key])
      }
      obj.save({useMasterKey:true})
      // logger.info(JSON.stringify(obj))
    }
  } else {
    logger.info('could not import data from '+url)
  }
})

and the export function on the other side:

// export data to other server
Moralis.Cloud.define('exportData', async (request) => {
  const draftQuery = new Moralis.Query(request.params.objectClass)
  const exportData = draftQuery.find({useMasterKey:true})
  return exportData
},{
  fields : ['objectClass'],
})

Don’t let them exposed!! Only temporarily uncomment them when you use them :slight_smile:

1 Like
#5

Very helpful - thank you @matiyin! Over time I suppose this could evolve into more of a scripted ETL process to allow moving tables and data between environments, such as from Prod<-> Test

#6

yeah what slows me down a lot in development is moving and cloning servers.

Sometimes you want to start fresh, especially when using several iterations of smart contracts the db tends to get full and messy.
But it’s a lot of steps (in my case) to set up from scratch. Cloud functions, ACL/CLP, plugins, event sync, scheduled jobs and maintenance jobs to add extra fields to the default Moralis structure…

Cloning with all settings would be amazing, with the option to include data or even selective data.

I think we had a brief thread on it last year and the team was talking about possibly adding native mongoDB cli support. Didn’t follow up if that’s possible now, although I saw some access keys were added to the Server Details screen.

But cloning all server setup without data would be more pressing, and that’s probably beyond mongo.

For the rest everything still e-ok :woman_mage: :mage:

#7

there is direct access to mongo db now: How to do database backups?

1 Like
#8

cool, can that be restored in another moralisdb? Or is the use limited?

Anyway https://roadmap.moralis.io/b/feature-requests/clone-server-with-all-settings-and-setups :nerd_face:

#9

I made a slight tweak in the export function to require a secret be passed along to help secure things until we get server clone…just in case I forget to comment it out :slight_smile:

// export data to other server
Moralis.Cloud.define(
  "exportData",
  async (request) => {
    const userSecret = request.params.secret;
    const dataSecret = await getImportExportSecret();
    if (userSecret !== dataSecret) throw "Incorrect password";
    const draftQuery = new Moralis.Query(request.params.objectClass);
    const exportData = draftQuery.find({ useMasterKey: true });
    return exportData;
  },
  {
    fields: ["objectClass", "secret"],
  }
);

async function getImportExportSecret() {
  const config = await Moralis.Config.get({ useMasterKey: true });
  return config.get("import_export_secret");
}
#10

I think that you can restore it in another moralis db

1 Like
#11

excellent choice thanks!