Hi everyone, recently, I have been worked on the couchbase with a request of data mirror from one couchbase database to another.
Basically, what I have done can be summarised as: we try to export original data and use a NodeJS script to generate a query for all the original data and finally run the query to inject all data from original data (One word: Using query to copy and paste data)
Here is the query generation script:
// This script can only be suitable for the data records which is less than 2000;
const fs = require("fs");
function convert(collectionName, sourceFile, query, resultFile) {
fs.readFile(sourceFile, "utf8", (err, data) => {
if (err) {
console.log("Source file read failed:", err);
return;
}
const json = JSON.parse(data);
const response = json.reduce((acc, cur, index) => {
const key = cur[collectionName].id.toString();
const value = cur[collectionName];
const valueString = JSON.stringify(value);
cur = ('("' + key + '", ' + valueString + ")").replace(/\\/g, "");
comma = index === json.length - 1 ? "" : ", ";
acc += cur + comma;
return acc;
}, "");
if (response) {
const final = `${query} ${response}`;
fs.writeFile(resultFile, final, function (err) {
if (err) return console.log(err);
console.log(`Generated data have been saved into ${resultFile}`);
});
}
});
}
Here is the example of how to do it:
Step 1: we need to go to the original data source we want to copy from (The couchbase bucket
-> collection
-> document
).
Just provide an example here, if I want to mirror data for test-bucket (bucket name) -> test-collection (collection name) -> test-documents (document name)
Step 2: we click the Query tab on the left side menu bar
Step 3: we run a query: SELECT * FROM test-bucket
.test-collection
.test-documents
Step 4: we select EXPORT
button on the top right corner and you will see a JSON file export modal window:
Step 5: we name the file name as test-documents
Then Click Save button to export data as JSON file
Step 6: You can open the convert.js
script I have attached below and you will see this setup from line 39 to line 45 below:
convert(
"test-documents", // the collection name
"./test-documents.json", // the file you have exported from CouchBase (Step 5)
"INSERT INTO `test-bucket`.`test-collection`.`test-documents`(KEY, VALUE) VALUES ", // the target bucket -> collection -> document
"test-documents-query.txt" // the final generated result query for data mirror / injection usage
);
Step 7: Modify the script provided in above (Step 6) based on your specific requirements.
As an example here, we want to copy the one couchbase database test-documents (document name) data to another couchbase database test-documents (document name), we can run above script for generating the data injection /mirror query.
run
node convert.js
command in your terminal within the current folder path and you will be able to generate the new query text file called as test-documents-query.txt
.
Step 8: We go to the destination / target document (eg: test-documents) and go to the Query tab (Same as Step 2) and paste the new generated query inside the Query Editor
Step 9 [Optional]: If you just created a new document for UK couchBase, you need to run this query first to avoid some index errors which will be reported by couchbase
CREATE PRIMARY INDEX ON `default`:`test-bucket`.`test-collection`.`test-documents`
*** If you already have document primary index setup already, you can skip Step 9. ***
Step 10: You will open the final generated data injection text file: test-documents-query.txt
and copy all the contents and paste into the Query Editor
Step 11: Click the Execute
button and starting injecting the data from one test-documents to another 🚀🚀
Some Additional Tips:
Before mirror the data, better to try to use the script locally (
localhost:3000
), so you will be able to get familiar with and understand this script before actually use itAlways ensure the bucket name, collection name and document name are entered correctly. Otherwise, you might inject the data in a wrong document
Finally, as a reference,
You might be able to find the work I have been working on:
https://github.com/DamengRandom/n1ql-query-generator
Thank you very much for reading, hopefully it could help a little with your couchbase data mirror process.
Top comments (0)