How to create an application that performs difficult tasks and show the result in an interface (for example, in the Google Search Console)

Question

Imagine this:

• I need to download an XML document from a URL;
• I have to develop this document and save its information in the database, creating or updating many rights.

I think the best way is to use queues. Or maybe I can also use cronjobs.

My problem is this: if I use the same application to perform heavy tasks, and also to show end users the results of these difficult tasks, it may happen that heavy tasks slow down the main site.

Take a more concrete example from real life: the Google Search Console (or any other application that performs heavy tasks and displays results to the end user).

The Google Search Console receives an XML map, then starts loading each web page, a large analysis is performed on each web page, and then the result is stored in a database, so the end user can see errors on his website and other useful information.

So, as a hypothesis, I want to re-create the Google Search Console as a Symfony application, which are possible approaches?

I think I should definitely use queues, but the application that loads the web pages and the application that processes these web pages and the public interface that shows the result of these operations is the same application or two or three separate applications

That is, should I create a unique application that performs all these actions, or am I creating an application to download web pages, another to process these web pages and another to show the user the results?

I think a lot about this and I cannot find a good design.

Because my istinct is to create multiple applications for each of these tasks in order to do the job, but it seems that creating multiple Symfony applications is not a good choice: Symfony 2 multiple applications?

So, I really don't know which path to follow: multiple applications or one large application? And if I use one large application, should I use cronjobs or am I still using queues?

Answer 1

• I will talk about architecture first :

2. You can make a Service , for example this :

namespace SomeBundle\Utils; use SomeBundle\Entity\MyEntity; use SomeBundle\myException; class SomeService { private $var; public function __construct() { } public function doStuff() { // Do stuff } } 

To debug your service, you can call it from testController or the main controller .

3. You can make a command like this :

 namespace SomeBundle\Command; use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand; use Symfony\Component\Console\Input\InputInterface; use Symfony\Component\Console\Output\OutputInterface; use SomeBundle\Utils\SomeService; class SomeCommand extends ContainerAwareCommand { protected function configure() { $this->setName('some:command') ->setDescription('My awesome command'); } protected function execute(InputInterface $input, OutputInterface $output) { $container = $this->getContainer(); $sevice = new SomeService($container); $results = $service->doStuff(); } } 

4. You can make a command application , for example this :

 require __DIR__.'/vendor/autoload.php'; require_once __DIR__.'/app/AppKernel.php'; $kernel = new AppKernel('dev', true); use SomeBundle\Command\SomeCommand; use Symfony\Bundle\FrameworkBundle\Console\Application; $application = new Application($kernel); $application->add(new SomeCommand()); $application->run(); 

Hope this helps!