Migrate a large dataset into a database — in batches

The scenario loads a large dataset in batches: it requests rows, cleans them, adds tags and stores them in the database on a loop until the data runs out.

  • Data moves in batches — memory never overflows
  • Every record is cleaned and tagged
  • Handles even very large datasets
  • The migration runs itself, no manual babysitting

How it works

To create records from signMe without manual input, use a ready-made scenario.

  1. Starts when: SubTrigger
  2. Then: HF_FetchRows
  3. Then: Extract_Rows
  4. Check: HasRows?
  5. If yes: Row_Splitter
  6. Then: Transform_RemoveId_AddMeta
  7. Then: Mongo_InsertOrUpsert
  8. Then: Aggregate
  9. Then: setOffset
  10. If no: NoRows_Offset
  11. Then: Config_Start
  12. Then: InsertBatch
  13. Check: ContinueLoop?
  14. Starts when: Trigger_Manual

You can launch this integration in Scriptera: describe the task in plain words — the scenario is built, launched and monitored for you.