Sean Sube
|
aefa5b4613
|
fix(api): clear job cancelled flag when worker starts a new job (#269)
|
2023-03-19 17:57:14 -05:00 |
Sean Sube
|
2e89fd43d3
|
fix(api): only remove running jobs from running state
|
2023-03-18 19:21:40 -05:00 |
Sean Sube
|
e5862d178c
|
fix(api): assume inversion tokens are embeddings for now
|
2023-03-18 18:35:11 -05:00 |
Sean Sube
|
1d52dc684d
|
init last progress on worker context
|
2023-03-18 17:27:41 -05:00 |
Sean Sube
|
e08a9aa2ab
|
add pending job list to pool
|
2023-03-18 17:26:28 -05:00 |
Sean Sube
|
8cbdad3a71
|
feat(api): add pending field to image ready response
|
2023-03-18 17:25:13 -05:00 |
Sean Sube
|
15b6e036e1
|
fix(api): maintain list of pending jobs
|
2023-03-18 17:15:18 -05:00 |
Sean Sube
|
588c8c7fdb
|
fix(api): track last progress within worker
|
2023-03-18 15:32:49 -05:00 |
Sean Sube
|
5106dd48a9
|
remove another ref to finished queue
|
2023-03-18 15:27:07 -05:00 |
Sean Sube
|
b026566ccb
|
remove remaining references to finished queue and worker
|
2023-03-18 15:26:19 -05:00 |
Sean Sube
|
d1565b056e
|
apply lint, make missing images an error
|
2023-03-18 15:16:41 -05:00 |
Sean Sube
|
7cf5554bef
|
feat(api): add error flag to image ready response
|
2023-03-18 15:13:42 -05:00 |
Sean Sube
|
aec540a524
|
feat(api): add server setting for CUDA memory limit (#211)
|
2023-03-18 13:40:37 -05:00 |
Sean Sube
|
226710a015
|
fix(api): use exception level logs
|
2023-03-16 22:29:07 -05:00 |
Sean Sube
|
4b832f3d8d
|
more lint, more trace
|
2023-03-16 20:22:20 -05:00 |
Sean Sube
|
c8c5e9f42e
|
fix(api): handle more out-of-memory errors in the workers
|
2023-03-16 18:34:28 -05:00 |
Sean Sube
|
b2eb406197
|
fix(api): handle CUDA memory errors in workers
|
2023-03-15 08:51:29 -05:00 |
Sean Sube
|
9555a7a3ea
|
lint(api): only log new worker message if some workers need to be restarted
|
2023-03-11 13:30:54 -06:00 |
Sean Sube
|
66c42485cb
|
feat(api): add support for extremely long prompts
|
2023-03-07 19:00:25 -06:00 |
Sean Sube
|
9d9bd1a639
|
apply lint
|
2023-03-07 08:02:53 -06:00 |
Sean Sube
|
35dc8a0bc4
|
improve exit logging
|
2023-03-05 21:37:39 -06:00 |
Sean Sube
|
c0a01efef4
|
fix(api): track currently active worker for each device
|
2023-03-05 21:28:21 -06:00 |
Sean Sube
|
57fed94337
|
fix(api): exit worker on memory allocation errors
|
2023-03-05 21:11:33 -06:00 |
Sean Sube
|
cb460a0c59
|
fix(api): add worker PID to log messages
|
2023-03-05 20:25:02 -06:00 |
Sean Sube
|
4ae4ce176c
|
fix(api): attempt to recycle leaking workers when a job finishes
|
2023-03-05 20:13:28 -06:00 |
Sean Sube
|
3a4928e59b
|
fix(api): prevent workers from blocking on their progress queues
|
2023-03-05 20:07:06 -06:00 |
Sean Sube
|
edc55ae8b4
|
fix(api): finished job notification should not block worker
|
2023-03-05 19:53:44 -06:00 |
Sean Sube
|
cfc20d3133
|
fix(api): improve cache logging
|
2023-03-05 19:30:52 -06:00 |
Sean Sube
|
39b9741b24
|
fix(api): show VRAM percent in logs
|
2023-03-05 19:23:23 -06:00 |
Sean Sube
|
7a3a81a4ef
|
fix(api): track and repeatedly attempt to recycle leaking workers (#219)
|
2023-03-05 18:58:13 -06:00 |
Sean Sube
|
1f3a5f6f3c
|
fix(api): track completed jobs for each device worker (#170)
|
2023-03-01 19:09:18 -06:00 |
Sean Sube
|
21fc7c5968
|
fix(api): mark all convert methods as no_grad
|
2023-03-01 08:26:40 -06:00 |
Sean Sube
|
c99aa67220
|
name threads, max queues, type/lint fixes
|
2023-02-28 21:44:52 -06:00 |
Sean Sube
|
c95ac1fbdd
|
avoid terminating workers because it breaks their queues
|
2023-02-28 08:53:17 -06:00 |
Sean Sube
|
0011f079d4
|
daemonize queue collectors
|
2023-02-28 06:55:15 -06:00 |
Sean Sube
|
cad0d37604
|
some pending queue logging
|
2023-02-27 23:43:38 -06:00 |
Sean Sube
|
4ae3d9caa2
|
remove task done
|
2023-02-27 23:18:37 -06:00 |
Sean Sube
|
7e0ccdb1af
|
remove pending queues after joining
|
2023-02-27 23:14:20 -06:00 |
Sean Sube
|
1ce98ace33
|
add value error handling
|
2023-02-27 23:12:53 -06:00 |
Sean Sube
|
da6ae5d62f
|
more logging around shutdown, close queues
|
2023-02-27 23:01:26 -06:00 |
Sean Sube
|
988088d64e
|
quit workers on keyboard signal
|
2023-02-27 22:52:43 -06:00 |
Sean Sube
|
953e5abd36
|
handle empty errors
|
2023-02-27 22:45:29 -06:00 |
Sean Sube
|
136759285d
|
set queue timeouts
|
2023-02-27 22:37:43 -06:00 |
Sean Sube
|
0793b61c3a
|
consistently pass job key to workers
|
2023-02-27 22:25:53 -06:00 |
Sean Sube
|
06f06f5a11
|
error handling in all threads
|
2023-02-27 19:48:51 -06:00 |
Sean Sube
|
113ad05293
|
typo
|
2023-02-27 17:36:26 -06:00 |
Sean Sube
|
2327b24022
|
join all threads
|
2023-02-27 17:35:31 -06:00 |
Sean Sube
|
66a20e60fe
|
run logger in a thread, clean up status
|
2023-02-27 17:14:53 -06:00 |
Sean Sube
|
13395933dc
|
always put progress in active jobs
|
2023-02-26 20:41:16 -06:00 |
Sean Sube
|
a37d1a4550
|
use progress queue
|
2023-02-26 20:37:22 -06:00 |
Sean Sube
|
401ee20526
|
fix finished flag
|
2023-02-26 20:13:16 -06:00 |
Sean Sube
|
525ee24e91
|
track started and finished jobs
|
2023-02-26 20:09:42 -06:00 |
Sean Sube
|
eb82e73e59
|
initialize list of finished jobs
|
2023-02-26 15:26:54 -06:00 |
Sean Sube
|
b931da1d2c
|
fix imports, lint
|
2023-02-26 15:21:58 -06:00 |
Sean Sube
|
85118d17c6
|
clear worker flags between jobs, attempt to record finished jobs again
|
2023-02-26 15:06:40 -06:00 |
Sean Sube
|
d1961afdbc
|
re-implement cancellation
|
2023-02-26 14:36:32 -06:00 |
Sean Sube
|
584dddb5d6
|
lint all the new stuff
|
2023-02-26 14:15:30 -06:00 |
Sean Sube
|
b880b7a121
|
set process titles, terminate workers
|
2023-02-26 13:09:24 -06:00 |
Sean Sube
|
6502e1e3c8
|
recycle worker pool after 10 jobs
|
2023-02-26 12:58:38 -06:00 |
Sean Sube
|
e0737e9e08
|
update progress and finished flag from worker
|
2023-02-26 12:51:11 -06:00 |
Sean Sube
|
f115326da7
|
apply patches within workers
|
2023-02-26 12:32:48 -06:00 |
Sean Sube
|
e1d0ad54b7
|
lock per worker, torch before ORT
|
2023-02-26 12:24:51 -06:00 |
Sean Sube
|
d765a6f01b
|
make logger start up well
|
2023-02-26 11:16:33 -06:00 |
Sean Sube
|
6998e8735c
|
rejoin worker pool
|
2023-02-26 10:47:31 -06:00 |
Sean Sube
|
943281feb5
|
wire up worker jobs
|
2023-02-25 23:55:30 -06:00 |
Sean Sube
|
f898de8c54
|
background workers, logger
|
2023-02-25 23:49:39 -06:00 |