1
0
Fork 0
Commit Graph

115 Commits

Author SHA1 Message Date
Sean Sube 5a517704ea
fix(tests): expand worker tests 2023-11-18 17:20:45 -06:00
Sean Sube d52c68d607
feat(api): add chain pipeline stage result type 2023-11-18 17:19:35 -06:00
Sean Sube e9b1375440
fix(api): write tests for embedding/inversion blending 2023-10-06 19:04:48 -05:00
Sean Sube 047e58c916
fix(api): continue adding tests, fix bugs encountered 2023-09-28 18:45:04 -05:00
Sean Sube 6fb0366f52
feat(api): make tile retries configurable 2023-09-11 18:41:59 -05:00
Sean Sube 404a314050
lint(api): add typed errors from cancelled job and download error 2023-09-04 21:20:04 -05:00
Sean Sube c506fe1e9f
make sure job is always defined, remove unused placeholders from logs 2023-07-17 22:46:02 -05:00
Sean Sube dfe9f6ee44
fix(api): split up worker name and job name to avoid logging false failures 2023-07-15 19:00:20 -05:00
Sean Sube 7f099c02e5
reset retries with each new job 2023-07-15 18:55:38 -05:00
Sean Sube ada482c183
add and use retry exception type 2023-07-15 17:05:27 -05:00
Sean Sube 1942c45789
apply lint 2023-07-15 16:55:52 -05:00
Sean Sube 7761aea28b
share retries throughout pipeline 2023-07-15 11:20:25 -05:00
Sean Sube 52cc4baaa4
fix circular import 2023-04-29 23:31:08 -05:00
Sean Sube 0573721d12
fix(api): always use accumulating progress counters 2023-04-29 23:28:51 -05:00
Sean Sube 7caaa9ed40
fix(api): correct dict types for 3.8 2023-04-24 17:39:06 -05:00
Sean Sube 30c96be24f
send token with admin requests, return worker status after restarting 2023-04-20 17:36:29 -05:00
Sean Sube cfdd926fff
fix(api): wait for worker to become idle before enqueueing next job (#286) 2023-04-15 20:37:53 -05:00
Sean Sube 1bfc7bee32
fix(api): run image workers as daemon procs 2023-04-10 23:23:08 -05:00
Sean Sube 9698e29268
lint(api): name context params consistently (#278) 2023-04-09 20:33:03 -05:00
Sean Sube 8e5971a293
fix lone else 2023-04-01 17:14:56 -05:00
Sean Sube 56c359c265
remove undefined names 2023-04-01 15:46:52 -05:00
Sean Sube 6bad599b3c
lint(api): move some chatty logs to trace level 2023-04-01 15:39:22 -05:00
Sean Sube f451d8db74
feat: add method parameter for highres mode 2023-04-01 14:26:45 -05:00
Sean Sube e0e09996de
fix(api): restart workers on HIP memory errors 2023-04-01 12:06:14 -05:00
Sean Sube 6aac0fe56b
fix(api): restart workers on MIOPEN memory errors 2023-04-01 11:59:03 -05:00
Sean Sube 0ea0442f02
apply lint 2023-03-26 18:24:22 -05:00
Sean Sube 4ddd69ba07
fix(api): watch for progress events from leaking workers 2023-03-26 18:22:16 -05:00
Sean Sube e1219cca90
fix(api): close queues after stopping workers 2023-03-26 18:21:53 -05:00
Sean Sube bb5d0635d1
sonar lint 2023-03-26 15:53:20 -05:00
Sean Sube 36bfccae09
fix(api): include worker totals in status endpoint 2023-03-26 15:49:43 -05:00
Sean Sube 2d2283e1eb
fix(api): attempt to read progress updates from recycled workers 2023-03-26 15:31:16 -05:00
Sean Sube 27500eccb5
fix(api): do not move jobs from pending to running until progress is reported 2023-03-26 14:22:59 -05:00
Sean Sube 0af406c47f
only enqueue jobs from progress worker 2023-03-26 14:02:57 -05:00
Sean Sube ccf8d51e08
feat(api): split up status endpoint by job status 2023-03-26 11:57:00 -05:00
Sean Sube ea36082e43
add job count to healthy worker logs 2023-03-26 11:53:06 -05:00
Sean Sube 8eab92a7df
define device on pending job 2023-03-26 11:49:58 -05:00
Sean Sube 83884bcafa
enqueue jobs on idle workers during progress check 2023-03-26 11:48:27 -05:00
Sean Sube 14ade83937
fix(api): enqueue next job when previous one finishes and after recycling worker 2023-03-26 11:41:45 -05:00
Sean Sube f3ab25f671
lint(api): add start method to worker pool 2023-03-26 11:30:07 -05:00
Sean Sube 2b179bebac
fix(api): always reset job counter when creating new device worker 2023-03-26 11:22:03 -05:00
Sean Sube 55e44e8ac9
fix(api): increment job counter for worker when it starts a new job (#283) 2023-03-26 11:18:27 -05:00
Sean Sube e552a5560f
feat(api): check device worker pool and recycle on a regular interval (#284) 2023-03-26 11:09:22 -05:00
Sean Sube aeb71ad50a
lint lock name 2023-03-26 08:30:34 -05:00
Sean Sube 95a61f3573
fix(api): restart worker threads when their respective queues are full 2023-03-25 13:46:12 -05:00
Sean Sube 88f4713e23
fix(api): use lock when restarting workers 2023-03-25 09:47:51 -05:00
Sean Sube 6b4c046867
pass pool to threads 2023-03-22 22:58:46 -05:00
Sean Sube 86c1b29c31
lint(api): extract worker thread main functions (#279) 2023-03-22 22:55:34 -05:00
Sean Sube 4dd68ea6b6
fix(api): restart worker threads if they crash 2023-03-22 19:58:46 -05:00
Sean Sube 0732058aa8
feat(api): detect Textual Inversion type from keys (#262) 2023-03-19 20:16:52 -05:00
Sean Sube aefa5b4613
fix(api): clear job cancelled flag when worker starts a new job (#269) 2023-03-19 17:57:14 -05:00