76 Commits

Author	SHA1	Message	Date
Sean Sube	1bfc7bee32	fix(api): run image workers as daemon procs	2023-04-10 23:23:08 -05:00
Sean Sube	6bad599b3c	lint(api): move some chatty logs to trace level	2023-04-01 15:39:22 -05:00
Sean Sube	0ea0442f02	apply lint	2023-03-26 18:24:22 -05:00
Sean Sube	4ddd69ba07	fix(api): watch for progress events from leaking workers	2023-03-26 18:22:16 -05:00
Sean Sube	e1219cca90	fix(api): close queues after stopping workers	2023-03-26 18:21:53 -05:00
Sean Sube	bb5d0635d1	sonar lint	2023-03-26 15:53:20 -05:00
Sean Sube	36bfccae09	fix(api): include worker totals in status endpoint	2023-03-26 15:49:43 -05:00
Sean Sube	2d2283e1eb	fix(api): attempt to read progress updates from recycled workers	2023-03-26 15:31:16 -05:00
Sean Sube	27500eccb5	fix(api): do not move jobs from pending to running until progress is reported	2023-03-26 14:22:59 -05:00
Sean Sube	0af406c47f	only enqueue jobs from progress worker	2023-03-26 14:02:57 -05:00
Sean Sube	ccf8d51e08	feat(api): split up status endpoint by job status	2023-03-26 11:57:00 -05:00
Sean Sube	ea36082e43	add job count to healthy worker logs	2023-03-26 11:53:06 -05:00
Sean Sube	83884bcafa	enqueue jobs on idle workers during progress check	2023-03-26 11:48:27 -05:00
Sean Sube	14ade83937	fix(api): enqueue next job when previous one finishes and after recycling worker	2023-03-26 11:41:45 -05:00
Sean Sube	f3ab25f671	lint(api): add start method to worker pool	2023-03-26 11:30:07 -05:00
Sean Sube	2b179bebac	fix(api): always reset job counter when creating new device worker	2023-03-26 11:22:03 -05:00
Sean Sube	55e44e8ac9	fix(api): increment job counter for worker when it starts a new job (#283)	2023-03-26 11:18:27 -05:00
Sean Sube	e552a5560f	feat(api): check device worker pool and recycle on a regular interval (#284)	2023-03-26 11:09:22 -05:00
Sean Sube	aeb71ad50a	lint lock name	2023-03-26 08:30:34 -05:00
Sean Sube	95a61f3573	fix(api): restart worker threads when their respective queues are full	2023-03-25 13:46:12 -05:00
Sean Sube	88f4713e23	fix(api): use lock when restarting workers	2023-03-25 09:47:51 -05:00
Sean Sube	6b4c046867	pass pool to threads	2023-03-22 22:58:46 -05:00
Sean Sube	86c1b29c31	lint(api): extract worker thread main functions (#279)	2023-03-22 22:55:34 -05:00
Sean Sube	4dd68ea6b6	fix(api): restart worker threads if they crash	2023-03-22 19:58:46 -05:00
Sean Sube	2e89fd43d3	fix(api): only remove running jobs from running state	2023-03-18 19:21:40 -05:00
Sean Sube	e5862d178c	fix(api): assume inversion tokens are embeddings for now	2023-03-18 18:35:11 -05:00
Sean Sube	e08a9aa2ab	add pending job list to pool	2023-03-18 17:26:28 -05:00
Sean Sube	8cbdad3a71	feat(api): add pending field to image ready response	2023-03-18 17:25:13 -05:00
Sean Sube	15b6e036e1	fix(api): maintain list of pending jobs	2023-03-18 17:15:18 -05:00
Sean Sube	588c8c7fdb	fix(api): track last progress within worker	2023-03-18 15:32:49 -05:00
Sean Sube	b026566ccb	remove remaining references to finished queue and worker	2023-03-18 15:26:19 -05:00
Sean Sube	d1565b056e	apply lint, make missing images an error	2023-03-18 15:16:41 -05:00
Sean Sube	7cf5554bef	feat(api): add error flag to image ready response	2023-03-18 15:13:42 -05:00
Sean Sube	aec540a524	feat(api): add server setting for CUDA memory limit (#211)	2023-03-18 13:40:37 -05:00
Sean Sube	226710a015	fix(api): use exception level logs	2023-03-16 22:29:07 -05:00
Sean Sube	4b832f3d8d	more lint, more trace	2023-03-16 20:22:20 -05:00
Sean Sube	9555a7a3ea	lint(api): only log new worker message if some workers need to be restarted	2023-03-11 13:30:54 -06:00
Sean Sube	9d9bd1a639	apply lint	2023-03-07 08:02:53 -06:00
Sean Sube	c0a01efef4	fix(api): track currently active worker for each device	2023-03-05 21:28:21 -06:00
Sean Sube	cb460a0c59	fix(api): add worker PID to log messages	2023-03-05 20:25:02 -06:00
Sean Sube	4ae4ce176c	fix(api): attempt to recycle leaking workers when a job finishes	2023-03-05 20:13:28 -06:00
Sean Sube	cfc20d3133	fix(api): improve cache logging	2023-03-05 19:30:52 -06:00
Sean Sube	39b9741b24	fix(api): show VRAM percent in logs	2023-03-05 19:23:23 -06:00
Sean Sube	7a3a81a4ef	fix(api): track and repeatedly attempt to recycle leaking workers (#219)	2023-03-05 18:58:13 -06:00
Sean Sube	1f3a5f6f3c	fix(api): track completed jobs for each device worker (#170)	2023-03-01 19:09:18 -06:00
Sean Sube	c99aa67220	name threads, max queues, type/lint fixes	2023-02-28 21:44:52 -06:00
Sean Sube	c95ac1fbdd	avoid terminating workers because it breaks their queues	2023-02-28 08:53:17 -06:00
Sean Sube	0011f079d4	daemonize queue collectors	2023-02-28 06:55:15 -06:00
Sean Sube	cad0d37604	some pending queue logging	2023-02-27 23:43:38 -06:00
Sean Sube	7e0ccdb1af	remove pending queues after joining	2023-02-27 23:14:20 -06:00