We all know very well, that automation can be scheduled the minimum-you-can-set once per an hour. This may satisfy mostly of automation we might build in our automation studio but sometimes we want to run automation again and again just as it finishes its previous run.
To achieve that we will create while loop with two automation where one automation is the main worker and does all the work you need it to do and the other will just call back the worker automation to start processing again.
This is very useful in many cases where you want to process Data Extensions with millions and millions of row and if you process all at once it will most likely end up into a run timeout.
Prerequisites
Add following columns to data extension that will be processed in batches:
- processed as Boolean
- processed_at as date
Worker automation
Main automation that will process Data extension with a lot of data will consists of following activities.
- query activity to take portion of data from the data extension you want to process in loop
- verification activity – to finish automation while loop when there is no data to process
- activities to process actual records taken into actual loop
- mark records as processed so they wont be processed again in the original data extension
- call the caller – this is script using to call other automation. For this you can find wast examples on the internet. You can achieve same with WSProxy or with platform function InvokePerform. Example can be found on stackexchange.
//InvokePerform simplified example
var obj = Platform.Function.CreateObject("Automation");
Platform.Function.SetObjectProperty(obj, "ObjectID", ObjectID);
var po = Platform.Function.CreateObject("PerformOptions");
var performResult = [0,0,0];
var performStatus = Platform.Function.InvokePerform(obj, "start", performResult, po);
//ws proxy sipmlified example
var WSProxy = new Script.Util.WSProxy();
WSProxy.performItem("Automation", { CustomerKey: customerKey }, "start");
Caller automation
Caller automation will consist of wait activity to make sure our worker automation is finished before we called it back to run again. And the script itself that runs the worker automation with method described earlier.
Good to know
Do not configure scheduler leave it blank. If you do so the automation could not be called from script
