Is there an easy way of getting over time the number of nodes allocated for Spark jobs/notebooks?
We can obviously report on individual jobs that are running (Monitor) and the number of CUs in use over time (Capacity Metrics), but one issue we're having is spark jobs grabbing a whole bunch of nodes and then causing the 'not enough nodes' error for other users. I'd like to both monitor node allocations as well maybe trigger alerts when we get close to the limit.
I'm already having to go on the hunt for rogue users - I'd like to catch it before it happens.
