Job Tracking Batch Notifications

Right now, we are using three nested foreach loops to get the information to run the package. However, I am sure that we could get the information using a single MySQL statement with joins and subqueries.

We have about 30 categories with 2000 users. Our goal is about 100 categories with 100,000 users, although it is obvious that foreach loops are not perfect (even now they take about a minute to work).

Circumstances: Users want to receive notifications if there is work available for trading that they can do in a specific area.

Purpose: Batch process (daily, weekly, etc.) Notifications to be sent to the Outbox

Technology: PHP, MySQL

What I still have:

Database:

"table.notification_options" : [id][user_id][category] "table.user" : [id][user_id][method_of_contact][contact_frequency][center_of_work_area_long][center_of_work_area_lat][distance_from_center] "table.work" : [id][post_date][longitude][latitude][category] 

Code:

 foreach user{ foreach category tracked{ foreach job in category posted <> $current_date-$batch_frequency{ if job inside workspace{ notify_user(job); } } } } 

The desired result is an array of job_ids arrays with user_id as the key [UserID] => {} jobs

eg.

  { [user1]{ job1, job4, job28 }, [user34]{ job3, job4, job34, job78 } { 

EDIT:

It’s a little more effective for me when I can select all tasks for one user. But for this, the foreach user is still needed.

  $category_id = get_category_from_notification_options($userid); $user_distance = get_user_work_distance($userid); "SELECT DISTINCT work.ID as workID, ( 6371 * acos( cos( radians(-46.409939) ) * cos( radians( jobs.lat ) ) * cos( radians( jobs.lng ) - radians(168.366180) ) + sin( radians(-46.409939) ) * sin( radians( jobs.lat ) ) ) ) AS distance FROM work,user WHERE work.categoryID == $category_id HAVING distance < $user_distance ORDER BY distance"; 
+4
source share
2 answers

I think you should do it the other way around to make it more efficient. Below I will show you the process that I used to create the request. So only the final request is what you need. But I explain the steps, so maybe this will help you in the future.

First, I would select all the tasks. Most likely, there are far fewer jobs than users if your goal is 100,000 users.

 select JOB.id, JOB.category FROM table.work JOB 

Now that we have all the tasks, let's see which users want to be notified about this.

 select JOB.id, JOB.category, NOTIFY.user_id FROM table.work JOB LEFT JOIN table.notification_options NOTIFY ON JOB.category=NOTIFY.category WHERE NOTIFY.user_id IS NOT NULL 

This creates a list for each job, all the identifier of the user who wants to be notified of this. I added a WHERE to remove all jobs from the list that no one wants to see. Now we can JOIN users table to get user information.

 select JOB.id , JOB.post_date , JOB.longitude , JOB.latitude , USR.user_id , USR.method_of_contact , USR.contact_frequency , USR.center_of_work_area_long , USR.center_of_work_area_lat , USR.distance_from_center , ((ACOS(SIN(USR.center_of_work_area_lat * PI() / 180) * SIN(JOB.latitude * PI() / 180) + COS(USR.center_of_work_area_lat * PI() / 180) * COS(JOB.latitude * PI() / 180) * COS((USR.center_of_work_area_long – JOB.longitude) * PI() / 180)) * 180 / PI()) * 60 * 1.1515) AS `distance` FROM table.work JOB LEFT JOIN table.notification_options NOTIFY ON JOB.category=NOTIFY.category LEFT JOIN table.user USR ON NOTIFY.user_id=USR.user_id WHERE NOTIFY.user_id IS NOT NULL HAVING `distance`<=USR.distance_from_center ORDER BY USR.user_id ASC, distance ASC 

I included the distance in the request. Please note that I am using HAVING to check if the distance is shorter than that provided by the user. If you add it to the WHERE , you will get the error message distance - unknown column. I also added the ORDER BY class to sort it first by user id and then by distance. This will make it easier to create the array you want in PHP.

Now there are many ways to implement daily / weekly intervals. One of them is to create separate scripts for each interval and select only those users who installed it. For example, you can create a script 'daily.php' that you run every day and have the following request

 select JOB.id , JOB.post_date , JOB.longitude , JOB.latitude , USR.user_id , USR.method_of_contact , USR.contact_frequency , USR.center_of_work_area_long , USR.center_of_work_area_lat , USR.distance_from_center , ((ACOS(SIN(USR.center_of_work_area_lat * PI() / 180) * SIN(JOB.latitude * PI() / 180) + COS(USR.center_of_work_area_lat * PI() / 180) * COS(JOB.latitude * PI() / 180) * COS((USR.center_of_work_area_long – JOB.longitude) * PI() / 180)) * 180 / PI()) * 60 * 1.1515) AS `distance` FROM table.work JOB LEFT JOIN table.notification_options NOTIFY ON JOB.category=NOTIFY.category LEFT JOIN table.user USR ON NOTIFY.user_id=USR.user_id WHERE NOTIFY.user_id IS NOT NULL AND USR.contact_frequency = 'daily' HAVING `distance`<=USR.distance_from_center ORDER BY USR.user_id ASC, distance ASC 

Now we have a request, let's create PHP code for it. We can iterate over all rows and create an array. Obviously, instead of creating an array, you can also directly process the result. Because if you create an array first, you need to loop around that array again.

 <?php $arNotify = array(); foreach ($queryresult as $row) { $userid = $row->user_id; $jobid = $row->id; //check if there is an entry for the user in the database, else create it if (!array_key_exists($userid, $arNotify)) $arNotify[$userid] = array(); //and then push the job $arNotify[$userid][] = $jobid; //the array is being created, but I still like to process the job directly //notify_user($userid, $jobid); } var_dump($arNotify); ?> 

There you go, an array, as you want, with jobs sorted by the nearest first.

+1
source

It seems to me that the distance you are taking is taken from the user table anyway (distance_from_center field?)

 SELECT DISTINCT ser.user_id, work.ID as workID, ( 6371 * acos( cos( radians(-46.409939) ) * cos( radians( jobs.lat ) ) * cos( radians( jobs.lng ) - radians(168.366180) ) + sin( radians(-46.409939) ) * sin( radians( jobs.lat ) ) ) ) AS distance FROM notification_options INNER JOIN jobs ON notification_options.category = jobs.category INNER JOIN user ON notification_options.user_id = user.user_id HAVING distance < user.distance_from_center ORDER BY distance 

EDIT. If you just need a list of tasks for each user remotely (which, if necessary, you can explode into an array for processing in php - although it is probably easier to use the above query to create an array), then you can use something like this : -

 SELECT user_id, GROUP_CONCAT(workID ORDER BY distance) FROM ( SELECT DISTINCT ser.user_id, work.ID as workID, ( 6371 * acos( cos( radians(-46.409939) ) * cos( radians( jobs.lat ) ) * cos( radians( jobs.lng ) - radians(168.366180) ) + sin( radians(-46.409939) ) * sin( radians( jobs.lat ) ) ) ) AS distance FROM notification_options INNER JOIN jobs ON notification_options.category = jobs.category INNER JOIN user ON notification_options.user_id = user.user_id HAVING distance < user.distance_from_center) Sub1 
+1
source

Source: https://habr.com/ru/post/1468895/


All Articles