Skip to content

Add possibliity to run a migration by batch #39

Description

@pp0rtal

Description

When we do a very long migration we can encounter already:

  • slowness: when the migration doesn't rely on an index (COLSCAN, IXSCAN)
  • non-verbose: we can't count (noCount:true is set) and we have no relative progress except the bulk log update once ready

In case there are no index, this is more annoying:

  • [resumability] a script will always take the same amount of time (scan is what is 99% of the processing)
  • [fatal] if there is a socket timeout configured (no match for a long period of time during the scan)

Proposed solution

Batch (avoid timeout, show progression)

We should allow to do the migration progressively relying _id index:

  • We iteratively fetch the ranges of _id: .find({_id:{$gte:<previous batch _id limit>}}).sort({_id:1}).skip(<batch size>).limit(1))
  • We then apply the migration on the range

Interface: add a new option:

new MongoBulkDataMigration({
    query: // unchanged, might container _id here /!\
    options: {
        throttle: // (already exisst) should throttle over ids
        batchScanSize: 100_000 // default: scan by 100K docs  
    }
})

A batch might result to 0 updates, that's intended.
Ideally, we'd prefer query already set _id to not be overridden, so we should cumulate both .

Resumability

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions