Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enqueued jobs do not respect queue priorities #360

Open
salmonsteak1 opened this issue Sep 24, 2024 · 4 comments · May be fixed by #410
Open

Enqueued jobs do not respect queue priorities #360

salmonsteak1 opened this issue Sep 24, 2024 · 4 comments · May be fixed by #410

Comments

@salmonsteak1
Copy link

salmonsteak1 commented Sep 24, 2024 •
edited
Loading

Hey there, I'm testing out the queue priorities in solid queue. My queue.yml is as follows:

default: &default
  dispatchers:
    - polling_interval: 1
      batch_size: 500
  workers:
    - queues: [urgent*, semi_urgent*, default*, low*]
      threads: 1
      processes: 1
      polling_interval: 0.1

development:
  <<: *default

test:
  <<: *default

production:
  <<: *default

I also have 3 test jobs for each queue priority level - A, B and C:

# Testing a low priority job A
module TestSqJobs
  class LowPriorityJobA < ActiveJob::Base
    self.queue_adapter = :solid_queue
    queue_as :low_A
    def priority
      0
    end

    def perform
      sleep(1)
    end
  end
end

Finally, I execute each job 5 times using the following ruby code:

job_classes = [
      TestSqJobs::SemiUrgentPriorityJobA,
      TestSqJobs::SemiUrgentPriorityJobB,
      TestSqJobs::SemiUrgentPriorityJobC,
      TestSqJobs::UrgentPriorityJobA,
      TestSqJobs::UrgentPriorityJobB,
      TestSqJobs::UrgentPriorityJobC,
      TestSqJobs::DefaultPriorityJobA,
      TestSqJobs::DefaultPriorityJobB,
      TestSqJobs::DefaultPriorityJobC,
      TestSqJobs::LowPriorityJobA,
      TestSqJobs::LowPriorityJobB,
      TestSqJobs::LowPriorityJobC
    ]

    job_classes.each do |job_class|
      5.times do |i|
        begin
          job_class.perform_later
          puts "Enqueued #{job_class} instance #{i + 1}"
        rescue ArgumentError => e
          puts "Skipping #{job_class} due to missing arguments: #{e.message}"
        end
      end
    end

    puts "Successfully enqueued 5 instances of each TestSqJobs job where possible."

I've observed this on mission control, and it seems like not all the jobs with the urgent prefix gets fulfilled first. Here's a screenshot of my observations:

image

We can see that jobs are taking from the default and semi_urgent queue instead of clearing out the job queues with the urgent prefix.
@rosa
Copy link
Member

rosa commented Sep 25, 2024

🤔 You're enqueuing the jobs sequentially, no? And the ones enqueued first are these:

      TestSqJobs::SemiUrgentPriorityJobA,
      TestSqJobs::SemiUrgentPriorityJobB,
      TestSqJobs::SemiUrgentPriorityJobC,

So they're going to be run before

      TestSqJobs::UrgentPriorityJobA,
      TestSqJobs::UrgentPriorityJobB,
      TestSqJobs::UrgentPriorityJobC,

because when they're enqueued, they're the only ones there.

@rosa
Copy link
Member

rosa commented Sep 25, 2024

Ah, I see what you mean. Yes, there's a bug there in the order of queues when they use prefixes. The order is the one returned from the DB from this query:

 relation.where(([ "queue_name LIKE ?" ] * prefixes.count).join(" OR "), *prefixes).distinct(:queue_name).pluck(:queue_name)

Which doesn't need to match the order in the LIKE clause. These should be reordered to match the order in the clause and the config.

This should work fine if you don't use prefixes, though (which is what I'd recommend in this case, anyway, collapsing all your low_, urgent_, etc. queues into one). I'll fix the case with prefixes.

@salmonsteak1
Copy link
Author

I see, thanks for the info @rosa! I was thinking to use wildcards together with the priority keywords so that I still can have separate queues for specific jobs while still having the benefit of using queue priorities.

Can I confirm what will be the scheduling algorithm used within jobs that match the same wildcard queue name? So, for example my queue.yml file would have queues defined as: [urgent*, semi_urgent*, default*, low*], and I have urgent_A, urgent_B and urgent_C jobs enqueued. Will the selection of these jobs be FCFS? Or would it be by random selection? Thanks!

@rosa
Copy link
Member

rosa commented Sep 25, 2024

Within the same wildcard name, urgent_A, urgent_B, urgent_C, the selection would be non-deterministic, it'll depend on the order the DB returns the queue names from SELECT queue_name FROM solid_queue_ready_executions WHERE queue_name LIKE 'urgent_%').

If you use exact queue names, polling will be faster, and the order will be deterministic.

@rosa rosa linked a pull request Nov 14, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix queue order when combining multiple prefixes or prefixes and names rails/solid_queue
2 participants
@rosa @salmonsteak1