Pentaho Maker: Create ETL Jobs Dynamically Based on User Input

Question

In my application, the user can specify the format of his file. Based on user input, we dynamically create an SSIS package.

http://lakshmik.blogspot.com/2005/05...eate-ssis.html

The dynamically created SSIS package is used to process user files.

We want to evaluate the Pentaho-Kettle for this requirement. Is it possible for Kettle to dynamically create ETL jobs based on user inputs?

If there is no Pentaho, is there any Java ETL tool that can be used to dynamically create ETL jobs?

Answer 1

I don’t know about others, but it’s traditionally quite difficult in a teapot, but people did it differently.

The best option for this is a step (new) that allows you to do really smart things with metadata - but it only works for some basic steps. I think he will do what you want, read about it on Matt Casters blog (PDI Creator and god) here:

http://www.ibridge.be/?s=inject&submit=Go

If this does not work; then your other options should follow the path of the common field name (nasty) or dynamically generate the conversion. It's easier than it sounds, but you will need to be much more involved in Java than usual for the ETL tool.

Answer 2

Perhaps not very difficult.

You can use the Kettle API to dynamically create transformations that can do everything Kettle does. The GUI designer uses the API to create the transforms, so anything you can do with the GUI you can do through the API.

If you look in the source tree "test", you will find many examples of how to dynamically create transforms.