Azure Data Lake Loop

Question

Does Azure Data Lake Analytics and U-SQL support While or For support for Loop and multiple exits? I want the output to be in multiple files using a single USQL execution.

This is what I want:

Foreach @day in @days @dataToSave = SELECT day AS day, company AS Company, FROM @data WHERE @day = @day @out = @day + ".txt" OUTPUT @dataToSave TO @out USING Outputters.Text(); Next 

I know that I can use powershell, but I think that it will cost the performance prior to execution.

Answer 1

U-SQL does not support While or For. You can use WHERE statements to filter the extracted data and virtual columns to filter based on paths / file names ( example ).

For output to multiple files, you can write a unique set of lines and a WHERE clause for each output, if its a reasonable number of files.

As you said, you can also script using Powershell or U-SQL ( example ).

Dynamic output for multiple files is currently limited to private previews. Please send usql email to microsoft dot com with your script if you are interested in this feature, as this might work for your script based on your description.

Hope this helps, and let me know if you have more questions about implementing any of these solutions.

Answer 2

Try this using also output:

 public override void Output(IRow input, IUnstructuredWriter output) { using (System.IO.StreamWriter streamWriter = new StreamWriter(address + _file, true)) //Save on file! } 

Answer 3

You can try to create a custom output and ignore the output file and write to your own file! public override void Output (IRow string, IUnstructuredWriter output)