Storm and Spring 4 Integrations

I have a prototype storm application that reads a STOMP stream and saves the output to HBase. It works, but not very flexible, and I try to configure it more consistently with our other applications, but I was not very able to figure out how the current mode works with Storm. We use the spring-jms classes, but instead of using them in the standard spring style, they are created at runtime and set dependencies manually.

This project: https://github.com/granthenke/storm-spring looked promising, but it was not affected after a couple of years and does not build properly, because the storm jars were taken into the Apache incubator and repackaged.

Is there something that I am missing, or is it not worth it to integrate these things?

+6
source share
4 answers

Actually, storm-spring seems to be what you are looking for, but it does not update and has limitations (cannot define tasks on bolts / nozzles, for example, etc.). Maybe you need to flip your own integration?

Do not forget your goal: a cluster with many employees. How does spring work when you deploy your topology with awi (like rebalance) to another employee? Does this mean that it must initiate a new spring context for the working JVM at startup before Storm expands the target bolts / spouts and defines the executors?

IMHO, if you only define Storm components in the spring configuration, it should work (the start configuration for the topology then storms only controls the objects), but if you rely on spring to control other components (it seems like with spring-jms), then this may become messy for topology reorganization, for example (singleton per worker / jvm? Or the whole topology?).

It's up to you to decide if this is worth worrying, my problem with spring configuration is that you easily forget the storm topology (it seems this is one JVM, but could be a lot more). Personally, I define my own singletones for each class loader (for example, a static finale or with a double check lock if I need delayed instanciation), since it does not hide complexity (intermediate level).

+5
source

@zenbeni answered this question, but I want to tell you about my implementation, it is difficult to make a nozzle / bolts like spring beans. But to use other spring spring beans inside your pins / bolts, you can declare a global variable, and in your execution method, checking the whtether variable is null or not. If it is zero, you should get a bean from the application context. Create a class that contains the beans initialization method if it is not already initialized. See the ApplicationContextAware interface for more information (reusing Spring bean).

Code example:

Bolt class:

public class Class1 implements IRichBolt{ Class2 class2Object; public void prepare() { if (class2Object== null) { class2Object= (Class2) Util .initializeContext("class2"); } } } 

Util class to initialize beans, if not already initialized:

 public class Util{ public static Object initializeContext(String beanName) { Object bean = null; try { synchronized (Util.class) { if (ApplicationContextUtil.getAppContext() == null) { ApplicationContext appContext = new ClassPathXmlApplicationContext("beans.xml"); bean = ApplicationContextUtil.getAppContext().getBean( beanName); } else { bean = ApplicationContextUtil.getAppContext().getBean( beanName); } } } catch (Exception e) { e.printStackTrace(); } return bean; } } 

Listener for changing application context:

 @Component public class ApplicationContextUtil implements ApplicationContextAware { private static ApplicationContext appContext; public void setApplicationContext(ApplicationContext applicationContext) throws BeansException { appContext = applicationContext; } public static ApplicationContext getAppContext() { return appContext; } } 

Note. Each worker will initialize the spring context as it works in different JVMs.

UPDATE

If you want to use the spring bean class in which you have previously assigned values, try this,

Note. Passing the current class to the Bolt constructor

A class (topology creation class) that already contains values:

 public class StormTopologyClass implements ITopologyBuilder, Serializable { public Map<String, String> attributes = new HashMap<String, String>(); TopologyBuilder builder=new TopologyBuilder(); builder.setBolt("Class1",new Class1(this)); builder.createTopology(); } 

Bolt using the single argument constructor:

 public class Class1 implements IRichBolt{ StormTopologyClass topology; public Class1 (StormTopologyClass topology) { this.topology = topology; } } 

Now you can use the variable attributes and values ​​in the bolt class.

+6
source

I understand that this is very important after that, but have you thought about using the Apache camel to handle JMS connections? Camel is not IOC or DI, but it models enterprise integration models. Maybe this is what you (were looking for?) Are looking for?

Nick.

+1
source

Perhaps this tutorial will help you.

http://spring.io/guides/gs/messaging-stomp-websocket/

-1
source

Source: https://habr.com/ru/post/971723/


All Articles