🤔[question] Get Slots per User

[Leon0402]

Describe your question

Hi,
is it possible to know for each user how many slots they use and ideally which tasks use these slots? Does determined cli support that?
There are different commands like det user, det task, det shell, but it is not obvious how to get the combined information from this.

Checklist

• Did you search the docs for a solution?
• Did you search github issues to find if somebody asked this question before?

[ioga]

Hello,

Today you'd probably have to do five commands det [experiment|notebook|tensorboard|shell|command] list and aggregate their output. they should all have --csv option which should make this easier to do programmatically.

[Leon0402]

@ioga Could you maybe give some more details?

I think there might be a misunderstanding here. None of these commands give me the slots used as far as I can see. E.g. when I do for instance det shell --all I get:

id   | username   | description   | state      | exitStatus   |resourcePool   | workspaceName

With some preprocessing I can filter for active shells here. But I don't see a way to get the slots.

Maybe to give a motivation why this is needed: Multiple people use the cluster and can create shells (and other stuff) with reserved slots (gpus). As we have a maximum number of gpus, it would be really nice to get an overview over how many gpus are used by each user. Currently we do this manually in an excel, but people forget to update that.

Edit: Another related topic, it would be really nice if the cli code would also return the csvs / json so they can be called programmatically easily. Right now they print to the terminal and have no return value. I used some mocking framework to get the output, but that is a little bit akward.

[ioga]

ah, so you also need the number of slots. in this case I am afraid you'll have to do det [experiment|notebook|tensorboard|shell|command] config $TASK_ID for each task, and extract the number of slots from that.

[Leon0402]

Would be nice as a feature request to include that information directly in the table. Both for the cli and the graphical userinterface. At least that is an information that would be beneficial to us.

[ioga]

yep I agree, and I've created an internal ticket to track this request.