Housing system or ESB?

Question

Recently I started working for a small company that is currently experiencing growing pains. I’m not sure what kind of system I’ll talk about now. In fact, we have a medley of many different applications that talk to each other using the built-in "integration system", which is a combination of SQL jobs, background services written in .NET, FTP transfers and SSIS, etc.

Here's a bird's eye view: Our public website is an order entry system (third-party shopping software) hosted outside the supplier. We upload order information every 4 hours a day. This data is then massed by our home-based “integration system”, which transfers this information to our inventory and warehouse management system (WMS). It also transfers information to MS Great Plains, Pulse, PayFuse and third-party CMS, etc.

As you might have guessed, this architecture is very fragile, and a small setback (for example, FTP crashes when the SQL query fails) may cause a discrepancy in the data to have a domino effect. There have been cases where, due to data-related problems or replication problems, it can lead to a shutdown of the entire warehouse, and we sometimes cannot accept orders, process or send orders.

My task is to redesign our systems and eliminate the rigid connection of systems to ensure business growth. What areas do I need to look into? I study ESB and SOA, but they tell me that my company cannot afford half a million dollars to go with iWay or Talend.

What are the options? Is internal development an answer and cheaper than implementing an ESB? Has anyone experienced such growing pains, and if so, how did you deal with integration?

Answer 1

This is how I would approach this problem.

• Forget the front design of one “perfect” system.
• Forget replacing everything at once.
• Find what causes a lot of pain, is relatively easy to replace and does not threaten the existence of a business. Work on it first.

In a way, the fact that there is a "potpourri of many different 3 applications" is good. You can leave the best in place by focusing on fixing those where there is the greatest value to the business.

Look for your stable business concepts and model them explicitly. Team and event models are your friends here. Group these concepts into Services, following SOA principles.

From your text, it looks like discussions around the SLA have already begun implicitly. Make these SLA discussions explicit, but focus on improving over time toward the goal, not nightly transformation.

The human infrastructure for this transformation is probably not worth the time, but spending 6 or 7 digits on a product before you know where you are going is not reasonable. Since you mentioned .Net, I used NServiceBus and found it to be a pleasant programming experience. You focus on your domain and your business logic and let NSB handle the plumbing / infrastructure. For low message throughput there is a free option. This allows you to deliver some value to the business before discussing budget and financing. There is a thriving NServiceBus community to help you get started in addition to the documentation on the website.

There are other options in the .net field as well, including MassTransit and EventStore . I personally haven’t used them and they are not functionally equivalent, so you will need to look at them and see what suits your needs and capabilities of your team.

Answer 2

Over the past couple of years, we have been working on some similar issues. Using an ESB will help you get rid of it. As soon as the teams begin to appreciate the denouement, it begins to give you, it will speed up the process. I am most familiar with NServiceBus, which, as we have discovered, has a very low barrier to entry. Developers quickly pick it up, and it is inexpensive. I would agree that you want to start with the area that is most important for the business, and first get it on solid ground.