Description
When we do a very long migration we can encounter already:
- slowness: when the migration doesn't rely on an index (COLSCAN, IXSCAN)
- non-verbose: we can't count (
noCount:true is set) and we have no relative progress except the bulk log update once ready
In case there are no index, this is more annoying:
- [resumability] a script will always take the same amount of time (scan is what is 99% of the processing)
- [fatal] if there is a socket timeout configured (no match for a long period of time during the scan)
Proposed solution
Batch (avoid timeout, show progression)
We should allow to do the migration progressively relying _id index:
- We iteratively fetch the ranges of _id:
.find({_id:{$gte:<previous batch _id limit>}}).sort({_id:1}).skip(<batch size>).limit(1))
- We then apply the migration on the range
Interface: add a new option:
new MongoBulkDataMigration({
query: // unchanged, might container _id here /!\
options: {
throttle: // (already exisst) should throttle over ids
batchScanSize: 100_000 // default: scan by 100K docs
}
})
A batch might result to 0 updates, that's intended.
Ideally, we'd prefer query already set _id to not be overridden, so we should cumulate both .
Resumability
Description
When we do a very long migration we can encounter already:
noCount:trueis set) and we have no relative progress except the bulk log update once readyIn case there are no index, this is more annoying:
Proposed solution
Batch (avoid timeout, show progression)
We should allow to do the migration progressively relying _id index:
.find({_id:{$gte:<previous batch _id limit>}}).sort({_id:1}).skip(<batch size>).limit(1))Interface: add a new option:
A batch might result to 0 updates, that's intended.
Ideally, we'd prefer query already set
_idto not be overridden, so we should cumulate both .Resumability