Sean Sube
|
7caaa9ed40
|
fix(api): correct dict types for 3.8
|
2023-04-24 17:39:06 -05:00 |
Sean Sube
|
30c96be24f
|
send token with admin requests, return worker status after restarting
|
2023-04-20 17:36:29 -05:00 |
Sean Sube
|
cfdd926fff
|
fix(api): wait for worker to become idle before enqueueing next job (#286)
|
2023-04-15 20:37:53 -05:00 |
Sean Sube
|
1bfc7bee32
|
fix(api): run image workers as daemon procs
|
2023-04-10 23:23:08 -05:00 |
Sean Sube
|
9698e29268
|
lint(api): name context params consistently (#278)
|
2023-04-09 20:33:03 -05:00 |
Sean Sube
|
8e5971a293
|
fix lone else
|
2023-04-01 17:14:56 -05:00 |
Sean Sube
|
56c359c265
|
remove undefined names
|
2023-04-01 15:46:52 -05:00 |
Sean Sube
|
6bad599b3c
|
lint(api): move some chatty logs to trace level
|
2023-04-01 15:39:22 -05:00 |
Sean Sube
|
f451d8db74
|
feat: add method parameter for highres mode
|
2023-04-01 14:26:45 -05:00 |
Sean Sube
|
e0e09996de
|
fix(api): restart workers on HIP memory errors
|
2023-04-01 12:06:14 -05:00 |
Sean Sube
|
6aac0fe56b
|
fix(api): restart workers on MIOPEN memory errors
|
2023-04-01 11:59:03 -05:00 |
Sean Sube
|
0ea0442f02
|
apply lint
|
2023-03-26 18:24:22 -05:00 |
Sean Sube
|
4ddd69ba07
|
fix(api): watch for progress events from leaking workers
|
2023-03-26 18:22:16 -05:00 |
Sean Sube
|
e1219cca90
|
fix(api): close queues after stopping workers
|
2023-03-26 18:21:53 -05:00 |
Sean Sube
|
bb5d0635d1
|
sonar lint
|
2023-03-26 15:53:20 -05:00 |
Sean Sube
|
36bfccae09
|
fix(api): include worker totals in status endpoint
|
2023-03-26 15:49:43 -05:00 |
Sean Sube
|
2d2283e1eb
|
fix(api): attempt to read progress updates from recycled workers
|
2023-03-26 15:31:16 -05:00 |
Sean Sube
|
27500eccb5
|
fix(api): do not move jobs from pending to running until progress is reported
|
2023-03-26 14:22:59 -05:00 |
Sean Sube
|
0af406c47f
|
only enqueue jobs from progress worker
|
2023-03-26 14:02:57 -05:00 |
Sean Sube
|
ccf8d51e08
|
feat(api): split up status endpoint by job status
|
2023-03-26 11:57:00 -05:00 |
Sean Sube
|
ea36082e43
|
add job count to healthy worker logs
|
2023-03-26 11:53:06 -05:00 |
Sean Sube
|
8eab92a7df
|
define device on pending job
|
2023-03-26 11:49:58 -05:00 |
Sean Sube
|
83884bcafa
|
enqueue jobs on idle workers during progress check
|
2023-03-26 11:48:27 -05:00 |
Sean Sube
|
14ade83937
|
fix(api): enqueue next job when previous one finishes and after recycling worker
|
2023-03-26 11:41:45 -05:00 |
Sean Sube
|
f3ab25f671
|
lint(api): add start method to worker pool
|
2023-03-26 11:30:07 -05:00 |
Sean Sube
|
2b179bebac
|
fix(api): always reset job counter when creating new device worker
|
2023-03-26 11:22:03 -05:00 |
Sean Sube
|
55e44e8ac9
|
fix(api): increment job counter for worker when it starts a new job (#283)
|
2023-03-26 11:18:27 -05:00 |
Sean Sube
|
e552a5560f
|
feat(api): check device worker pool and recycle on a regular interval (#284)
|
2023-03-26 11:09:22 -05:00 |
Sean Sube
|
aeb71ad50a
|
lint lock name
|
2023-03-26 08:30:34 -05:00 |
Sean Sube
|
95a61f3573
|
fix(api): restart worker threads when their respective queues are full
|
2023-03-25 13:46:12 -05:00 |
Sean Sube
|
88f4713e23
|
fix(api): use lock when restarting workers
|
2023-03-25 09:47:51 -05:00 |
Sean Sube
|
6b4c046867
|
pass pool to threads
|
2023-03-22 22:58:46 -05:00 |
Sean Sube
|
86c1b29c31
|
lint(api): extract worker thread main functions (#279)
|
2023-03-22 22:55:34 -05:00 |
Sean Sube
|
4dd68ea6b6
|
fix(api): restart worker threads if they crash
|
2023-03-22 19:58:46 -05:00 |
Sean Sube
|
0732058aa8
|
feat(api): detect Textual Inversion type from keys (#262)
|
2023-03-19 20:16:52 -05:00 |
Sean Sube
|
aefa5b4613
|
fix(api): clear job cancelled flag when worker starts a new job (#269)
|
2023-03-19 17:57:14 -05:00 |
Sean Sube
|
2e89fd43d3
|
fix(api): only remove running jobs from running state
|
2023-03-18 19:21:40 -05:00 |
Sean Sube
|
e5862d178c
|
fix(api): assume inversion tokens are embeddings for now
|
2023-03-18 18:35:11 -05:00 |
Sean Sube
|
1d52dc684d
|
init last progress on worker context
|
2023-03-18 17:27:41 -05:00 |
Sean Sube
|
e08a9aa2ab
|
add pending job list to pool
|
2023-03-18 17:26:28 -05:00 |
Sean Sube
|
8cbdad3a71
|
feat(api): add pending field to image ready response
|
2023-03-18 17:25:13 -05:00 |
Sean Sube
|
15b6e036e1
|
fix(api): maintain list of pending jobs
|
2023-03-18 17:15:18 -05:00 |
Sean Sube
|
588c8c7fdb
|
fix(api): track last progress within worker
|
2023-03-18 15:32:49 -05:00 |
Sean Sube
|
5106dd48a9
|
remove another ref to finished queue
|
2023-03-18 15:27:07 -05:00 |
Sean Sube
|
b026566ccb
|
remove remaining references to finished queue and worker
|
2023-03-18 15:26:19 -05:00 |
Sean Sube
|
d1565b056e
|
apply lint, make missing images an error
|
2023-03-18 15:16:41 -05:00 |
Sean Sube
|
7cf5554bef
|
feat(api): add error flag to image ready response
|
2023-03-18 15:13:42 -05:00 |
Sean Sube
|
aec540a524
|
feat(api): add server setting for CUDA memory limit (#211)
|
2023-03-18 13:40:37 -05:00 |
Sean Sube
|
226710a015
|
fix(api): use exception level logs
|
2023-03-16 22:29:07 -05:00 |
Sean Sube
|
4b832f3d8d
|
more lint, more trace
|
2023-03-16 20:22:20 -05:00 |