IPFS-BULK-UPLOAD project 'toString' error

Technical Issue Reporting
#22

use this to get the dynamic number of total files in a folder

let totalFiles = fs.readdirSync(folder).length;

#23

Hey, Iā€™ve been troubleshooting this string issue since yesterday with no success.
I have checked the names of files and are paddedhex 64 digit forms, I have also made sure the amount of them was between 5-10mb, even played with taking some out and making it less than 5mb. I am using pngā€™s.
I use the same folder structures. The substr(ā€œ64ā€) is deprecated
Aside from this deprecation, I have the same code as Filip. Of course I have my api key in there.
Could my server online be the issue?

Below is what I type and itā€™s response.

Node.js v17.4.0
bobo@bobos-Macbook-Air ipfsuploadbulk % node image.js
/Users/bobo/Downloads/ipfsuploadbulk/images.js:15
content: data.toString(ā€œbase64ā€)

TypeError: cannot read properties of undefined (reading ā€˜toStringā€™)
at ReadFileContext.callback (/Users/bobo/Downloads/ipfsuploadbulk/images.js:15:31

this is my code
let fs = require(ā€œfsā€);
let axios = require(ā€œaxiosā€);

let ipfsArray = [];
let promises = [];

for (let i = 0; i < 100; i++) {
let paddedHex = (ā€œ0000000000000000000000000000000000000000000000000000000000000000ā€ + i.toString(16)).substr("-64");

promises.push(new Promise( (res, rej) => {
fs.readFile(${__dirname}/export/${paddedHex}.png, (err, data) => {
if(err) rej();
ipfsArray.push({
path: images/${paddedHex}.png,
content: data.toString(ā€œbase64ā€)
})
res();
})
}))
}

Promise.all(promises).then( () => {
axios.post(ā€œhttps://deep-index.moralis.io/api/v2/ipfs/uploadFolderā€,
ipfsArray,
{
headers: {
ā€œEGLpjxRDz9WyGrI1gL18lalA0TxDS2pJKN7eF5gZfVGiXUdCM2k059r4g1tTaQ4Oā€: ā€˜ā€™,
ā€œContent-Typeā€: ā€œapplication/jsonā€,
ā€œacceptā€: ā€œapplication/jsonā€
}
}
).then( (res) => {
console.log(res.data);
})
.catch ( (error) => {
console.log(error)
})
})

any help would be great. thank you

#24

can you try with 1MB for total folder size?
after that youā€™ll have to check if files are in the expected folder (fs.readFile(${__dirname}/export/${paddedHex}.png, ()

#25

yes. thank you.
Not sure what is upā€¦ trying many things on other posts and changing directories to get the correct one, even tho i know the .pngā€™s are where they need to beā€¦
I got this error, something new after playing
bobo@bobos-MacBook-Air ipfsuploadbulk % node images.js
Debugger attached.
[Error: ENOENT: no such file or directory, open ā€˜/export/0000000000000000000000000000000000000000000000000000000000000000.pngā€™] {
errno: -2,
code: ā€˜ENOENTā€™,
syscall: ā€˜openā€™,
path: ā€˜/export/0000000000000000000000000000000000000000000000000000000000000000.pngā€™
}
[Error: ENOENT: no such file or directory, open ā€˜/export/0000000000000000000000000000000000000000000000000000000000000001.pngā€™] {
errno: -2,
code: ā€˜ENOENTā€™,
syscall: ā€˜openā€™,
path: ā€˜/export/0000000000000000000000000000000000000000000000000000000000000001.pngā€™
}
[Error: ENOENT: no such file or directory, open ā€˜/export/0000000000000000000000000000000000000000000000000000000000000002.pngā€™] {
errno: -2,
code: ā€˜ENOENTā€™,
syscall: ā€˜openā€™,
path: ā€˜/export/0000000000000000000000000000000000000000000000000000000000000002.pngā€™
}

#26

From that error it looks like it tries to search in an absolute path starting with /export

1 Like
#27

Actually figured out that it had to do with the amount of zeros in the filenameā€¦ I counted them many times, to all total 64, but I guess I donā€™t know how to count that high lmao. I just copied the zeros from inside the code to the images, with the corresponding 1, 2, 3ā€™s afterā€¦
wow. I definitely learned a lot troubleshooting this script tho. Thanks again all comments above

1 Like
#28

cryptokid, do you know if there is a workaround to the filesize of the folder? I mean this code is great but even have a folder that is 7mb will not work. Seems I can only upload about 10 files at a time. I did an upgrade with moralis and still cannot seem to do an actual BULK upload. Any insight would be great. Thanks again

#29

for now I donā€™t know of a solution with Moralis, you could try to upload all the files to an IPFS folder with another IPFS provider

#30

tried every solution on hereā€¦ the i = 0 one worked to file one file but overall nothingā€¦ extremely frustrated with this

#31

most of the times it tries to access a path that doesnā€™t exist

#32

data: { message: ā€˜Invalid files or path are duplicatedā€™ }

iā€™m getting this error now ā€¦ before it was just the toString error
any ideas ? full code below
headers: [Object],
transformRequest: [Array],
transformResponse: [Array],
timeout: 0,
adapter: [Function: httpAdapter],
xsrfCookieName: ā€˜XSRF-TOKENā€™,
xsrfHeaderName: ā€˜X-XSRF-TOKENā€™,
maxContentLength: -1,
maxBodyLength: -1,
validateStatus: [Function: validateStatus]
},
request: ClientRequest {
_events: [Object: null prototype],
_eventsCount: 7,
_maxListeners: undefined,
outputData: [],
outputSize: 0,
writable: true,
_last: true,
chunkedEncoding: false,
shouldKeepAlive: false,
_defaultKeepAlive: true,
useChunkedEncodingByDefault: true,
sendDate: false,
_removedConnection: false,
_removedContLen: false,
_removedTE: false,
_contentLength: null,
_hasBody: true,
_trailer: ā€˜ā€™,
finished: true,
_headerSent: true,
socket: [TLSSocket],
connection: [TLSSocket],
_header: ā€˜POST /api/v2/ipfs/uploadFolder HTTP/1.1\r\nā€™ +
ā€˜Accept: application/json\r\nā€™ +
ā€˜Content-Type: application/json\r\nā€™ +
ā€˜X-API-KEY: C1HjXxEKVTOJMLaqiB69wCJArzOgq20MZ1LASAY9giVTGDt3nn9KwRIixPvMVljs\r\nā€™ +
ā€˜User-Agent: axios/0.20.0\r\nā€™ +
ā€˜Content-Length: 4254937\r\nā€™ +
ā€˜Host: deep-index.moralis.io\r\nā€™ +
ā€˜Connection: close\r\nā€™ +
ā€˜\r\nā€™,
_keepAliveTimeout: 0,
_onPendingData: [Function: noopPendingOutput],
agent: [Agent],
socketPath: undefined,
method: ā€˜POSTā€™,
insecureHTTPParser: undefined,
path: ā€˜/api/v2/ipfs/uploadFolderā€™,
_ended: true,
res: [IncomingMessage],
aborted: false,
timeoutCb: null,
upgradeOrConnect: false,
parser: null,
maxHeadersCount: null,
reusedSocket: false,
host: ā€˜deep-index.moralis.ioā€™,
protocol: ā€˜https:ā€™,
_redirectable: [Writable],
[Symbol(kCapture)]: false,
[Symbol(kNeedDrain)]: false,
[Symbol(corked)]: 0,
[Symbol(kOutHeaders)]: [Object: null prototype]
},
data: { message: ā€˜Invalid files or path are duplicatedā€™ }
},
isAxiosError: true,
toJSON: [Function: toJSON]

#33

it looks like this is a new error that we have, we will have to fix it, this looks like is not from your code

#34

: (
any other way to upload nfts in bulk ?

#35

you can upload them separately with pinata if you want
I donā€™t know exactly how fast we will fix this ā€˜Invalid files or path are duplicatedā€™ error

#36

ill check it out thanksā€¦

#37

that error with Invalid files or path are duplicated should be fixed now

#38

wow lets sees if this will work

#39

its working but only downloading one file ā€¦ maybe the amount of zeros is wrong ā€¦ its 64 zeros and then 1 right ?

#40

I think that in total it has to be 64 characters, as in 63 zeros and 1

#41

ok let me fix it and come back thanks though regardlessā€¦ this is the fastest i received a reply for an issue ever ā€¦