Cannot upload .zip file to IPFS

Technical Issue Reporting
#1

I am attempting to write a simple ipfs backup script, which given a directory and a time period will sync that folder to ipfs. I also plan to add some additional logic so that the system wont backup something that it already has.

I am able to zip and read the folder passed in, but when it attempts to instantiate the Moralis.File I get the following error:

(node:12488) UnhandledPromiseRejectionWarning: TypeError: Cannot create a Parse.File with that data.
at new ParseFile (D:\Software Projects\Repos\ipfs-backup\node_modules\moralis\lib\node\ParseFile.js:203:15)
at backup (D:\Software Projects\Repos\ipfs-backup\backup-service.js:33:28)
at Object. (D:\Software Projects\Repos\ipfs-backup\backup-script.js:14:1)
at Module._compile (internal/modules/cjs/loader.js:1063:30)
at Object.Module._extensions…js (internal/modules/cjs/loader.js:1092:10)
at Module.load (internal/modules/cjs/loader.js:928:32)
at Function.Module._load (internal/modules/cjs/loader.js:769:14)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:72:12)
at internal/main/run_main_module.js:17:47

This is my code:

const compressDir = (directory) => {
    const dirPath = directory[0] === '/' ? directory.substring(1) : directory;
    const splitDirPath = dirPath.split('/');
    const lastDir = splitDirPath[splitDirPath.length - 1];
    const tempStoragePath = `./temp/${lastDir}.zip`;
    compressing.zip.compressDir(directory, tempStoragePath).then((x) => console.log(x))
                                                           .catch((err) => console.error(err));
    return tempStoragePath;
};

const backup = async (directory) => {
    const compressedDir = compressDir(directory);
    const backupDate = new Date();
    const zipFile = fs.readFileSync(compressedDir);
    console.log('zip', zipFile)
    // Perform encryption
    const moralisZipFile = new Moralis.File(`Backup ${backupDate}`, zipFile, 'application/zip');
    await moralisZipFile.saveIPFS();
    const backupLink = moralisZipFile.ipfs();

    const mailOptions = {
        from: email,
        to: email,
        subject: `Backup completed - ${compressedDir} - ${backupDate}`,
        text: `Backup - ${compressedDir} - ${backupDate}\n${backupLink}`
    };

    transporter.sendMail(mailOptions, (err, info) => console.log(err ? err : `Backup email sent: ${info.response}`));
    fs.unlink(compressedDir, (err) => {
        if (err) throw err;
        console.log(`${compressedDir} was deleted`)
    });
};

This is how I am importing Moralis, as I am using the node environment:
const Moralis = require('moralis/node');

#2

You could try to convert the file contents to base64

#3

I have tried changing the line that reads the file to encode to base64:

   const zipFile = fs.readFileSync(compressedDir, { encoding: 'base64' });

I have also tried encoding the file when declaring the Moralis.File:

    const base64 = "V29ya2luZyBhdCBQYXJzZSBpcyBncmVhdCE=";
    const moralisZipFile = new Moralis.File(`Backup ${backupDate}`, zipFile, { base64: base64 });

But the error is still the same.

#4

you can try this:

#5

This seems to work but I am now getting the following error:

(node:10264) UnhandledPromiseRejectionWarning: Error: File upload by public is disabled.

I have looked around but I am unsure whether I need to include some wallet authentication code in order to use the ipfs gateway, as ideally I do not want to do this.

#6

you can use a dummy user with username and password, authenticate with that user and then use IPFS

#7

What syntax would I need to use for that?

#8

https://docs.moralis.io/moralis-server/users/email-login#log-in-with-username

#9

Would using the master key also achieve this?

#10

I don’t know, you can try

#11

It seems to work using the master key.

I have implemented the btoa example in the link you provided. Now the system is working, but when it redownloads the zip file (I want to do this to verify the data isn’t corrupted), an error that says the following is outputted:

The archive is either in unknown format or damaged.

This is my current code:

const btoa = function(str){ return Buffer.from(str).toString('base64'); }

const compressDir = async (directory) => {
    const dirPath = directory[0] === '/' ? directory.substring(1) : directory;
    const splitDirPath = dirPath.split('/');
    const lastDir = splitDirPath[splitDirPath.length - 1];
    const tempStoragePath = `./temp/${lastDir}.zip`;
    await compressing.zip.compressDir(directory, tempStoragePath).then((x) => console.log(x))
                                                           .catch((err) => console.error(err));
    return tempStoragePath;
};

const backup = async (directory) => {
    const compressedDir = await compressDir(directory);
    const backupDate = new Date();
    const zipFile = fs.readFileSync(compressedDir, 'base64');
    // Perform encryption
    const moralisZipFile = new Moralis.File(`Archive.zip`, { base64: btoa(zipFile) });
    await moralisZipFile.saveIPFS({ useMasterKey: true });
    console.log('saved to ipfs')
    const backupLink = moralisZipFile.ipfs();
    console.log(backupLink)

    axios.get(backupLink).then((x) => {
        const { data } = x;
        console.log(typeof data)
        fs.writeFileSync('Test.zip', data);
    })

    fs.unlink(compressedDir, (err) => {
        if (err) throw err;
        console.log(`${compressedDir} was deleted`)
    });
}
#12

can you compare the file that you uploaded to IPFS with the file that you download from IPFS? to see if they are identical

#13

The files don’t appear to be identical. I’m not sure if there is anything I need to do to the data after downloading before I save it as a zip file.

#14

I realised that I needed to specify the encoding when writing the file, as it defaults to UTF-8.

fs.writeFileSync(`Backup-${backupDate}.zip`, x.data, { encoding: 'base64' })
#15

you should write it in binary somehow without converting it to text