How to implement loops in U-SQL

Question

Is it possible to implement a (while / for) loop in U-SQL without using C #. If not, can anyone use C # syntax to implement loops in u-sql.

I am extracting files from a specific date to a date, but right now I am extracting it by writing the file path manually.

DROP VIEW IF EXISTS dbo.ReadingConsolidated; CREATE VIEW IF NOT EXISTS dbo.ReadingConsolidated AS EXTRACT ControllerID int?, ParameterID int?, MeasureDate DateTime, Value float FROM "adl://eclwpsdatalake.azuredatalakestore.net/2015/7/1/Reading.csv", "adl://eclwpsdatalake.azuredatalakestore.net/2015/7/2/Reading.csv", "adl://eclwpsdatalake.azuredatalakestore.net/2015/7/3/Reading.csv", "adl://eclwpsdatalake.azuredatalakestore.net/2015/7/4/Reading.csv", "adl://eclwpsdatalake.azuredatalakestore.net/2015/7/5/Reading.csv", "adl://eclwpsdatalake.azuredatalakestore.net/2015/7/6/Reading.csv", "adl://eclwpsdatalake.azuredatalakestore.net/2015/7/7/Reading.csv" 

Note: these files are present in different folders.

Can a loop be possible over what?

Answer 1

The correct way to do this is to use virtual columns , and then rely on the elimination of partitions , so that in fact only files matching predicates are read (you can confirm that in the graph).

 CREATE VIEW IF NOT EXISTS dbo.ReadingConsolidated AS EXTRACT ControllerID int?, ParameterID int?, MeasureDate DateTime, Value float, date DateTime FROM "adl://eclwpsdatalake.azuredatalakestore.net/{date:yyyy}/{date:M}/{date:d}/Reading.csv"; @res = SELECT * FROM dbo.ReadingConsolidated WHERE date BETWEEN DateTime.Parse("2015/07/01") AND DateTime.Parse("2016/07/07");