What are priority and fairshare?
In a shared system like our central cluster, jobs get submitted to a queue and run within a particular order. The order is determined by the numerical priority of the job. This priority is determined using a multi factor priority algorithm which can take into account the size of the job, the length of time in the queue, the user or group that submitted it, the qos it was submitted to, etc. The vast majority of priority in our cluster comes from fairshare.
Fairshare is a method of allocating shares of a cluster to different groups and attempting to run jobs on the cluster to get them their “fair share”. The share is determined within a window of time with decreasing impact the further back in time jobs were run determined by a half life.
What does this mean in practice?
If your group has run a lot of jobs recently and exceed their “fair share”, the priority on submitted jobs for the group will go down. If your group has not used the cluster much within the window, your priority will go up. The scheduler takes in to account a few other factors into the priority calculation, but with much lower impact. In short, the idea is to try and keep a level playing field for all groups.
How are shares setup on the central cluster?
Shares of the cluster are determined by group size and investment in the cluster.
Each user by default gets 10 shares. The aggregate of the Fairshare “shares” gets assigned to the group.
5 users in a group - 5 x 10 = 50
Total group fairshare = 50
- Each group that invests in the cluster gets 1 fairshare allocation per $1000 invested (minimum $100k investment). Individual users get a baseline fairshare of 10 shares allocated to their group the investment allocations are added to the user allocations.
- 5 users in a group – 5 x 10 = 50
- $100K invested – $100,000/1,000 = 100
- Total group fairshare – 100 + 50 = 150