Hardening the computational backend

[mguidon]

There are a few issues in the computational backend that block us from scaling up. Jobs can get stuck in the queue and ec2 instances are not terminated properly.

GOAL: User can scale up to dozens of ec2 machines of different types.

In addition the Task Manager in the sim4life application can be improved upon (see #1731)

This reflects the following PO requirement:

• submitting jobs to the comp backend, monitoring jobs
• receiving computed jobs and performing postprocessing

---

MartinKippenberger

Beta Give feedback

1. Autoscaling: scaling down unused machines while asking for other types of machines does not work osparc-simcore#6367
   a:autoscaling computational clusters +2
   
2. Computational cluster: sometimes a cancelled job is not removed from the cluster osparc-simcore#6400
   a:director-v2 computational clusters +2
   
3. Clusters-keeper: cannot cope with long custom scripts osparc-simcore#6249
   High Priority a:clusters-keeper computational clusters +3
   
4. 🎨Clusters keeper/use ssm (🚨change in private clusters) osparc-simcore#6361
   a:clusters-keeper computational clusters +2
   

Tasks

Beta Give feedback

1. ✨Director-v2 computational scheduler: Persist mark for cancellation of runs osparc-simcore#6653
   a:director-v2 +1
   
2. Refactoring of in-house scheduling of computational jobs to allow scaling of simcore services and reduce load on system osparc-simcore#6666
   a:director-v2 +1
   
3. Provide logs of private clusters autoscaling status osparc-simcore#6667
   a:autoscaling a:infra+ops +2
   
4. Provide a way to show logs of a project without opening the project osparc-simcore#6669
   a:frontend a:webserver +2
   
5. Computational clusters: User needs visual feedback on the state of the computational clusters (# workers, running state, ...) osparc-simcore#5000
   a:autoscaling a:dask-service computational clusters t:enhancement +4
   
6. Autoscaling configuration issues: when the EC2_INSTANCES_KEY_NAME is changed, then the next autoscaling instance loses track of old instances osparc-simcore#4722
   a:autoscaling t:enhancement +2
   
7. Computational clusters: Allow to define cluster concurrency osparc-simcore#5074
   1 of 3
   a:autoscaling a:frontend a:webserver computational clusters t:enhancement +5
   
8. Autoscaling: Allow to set specific node labels based on EC2 type osparc-simcore#6043
   a:autoscaling +1
   
9. Improve make update-repo script with checking services in registry osparc-simcore#6394
   a:autoscaling t:maintenance +2
   
10. Autoscaling: Sometimes unlabelled docker nodes remain in the swarm forever osparc-simcore#6141
    a:autoscaling bug +2