47 Commits

Author	SHA1	Message	Date
Sean Sube	2eb90ba559	feat: better error messaging	2024-01-08 22:14:32 -06:00
Sean Sube	9b5d00a66a	move progress state to worker context	2024-01-05 20:12:41 -06:00
Sean Sube	4dc251cf4a	fix callback access	2024-01-04 19:39:44 -06:00
Sean Sube	b6da935be6	use progress type in command	2024-01-04 19:09:52 -06:00
Sean Sube	bdefaec0d2	serialize metadata	2024-01-03 23:27:40 -06:00
Sean Sube	8ae6069ac3	handle 0 progress	2024-01-03 23:19:56 -06:00
Sean Sube	7a5e6ab895	pass result on to command	2024-01-03 23:17:35 -06:00
Sean Sube	c4b831fe5c	fix typos	2024-01-03 21:45:13 -06:00
Sean Sube	9b5e894898	avoid waiting for final progress	2024-01-03 21:39:19 -06:00
Sean Sube	28fc2082c7	track results after each stage	2024-01-03 21:31:41 -06:00
Sean Sube	4f1bc84fd9	initialize fields	2024-01-03 21:20:46 -06:00
Sean Sube	0c504e3f69	report stage and tile count in progress	2024-01-03 21:16:44 -06:00
Sean Sube	44a8d61082	feat: add batch endpoints for cancel and status, update responses	2024-01-03 19:09:18 -06:00
Sean Sube	d52c68d607	feat(api): add chain pipeline stage result type	2023-11-18 17:19:35 -06:00
Sean Sube	047e58c916	fix(api): continue adding tests, fix bugs encountered	2023-09-28 18:45:04 -05:00
Sean Sube	6fb0366f52	feat(api): make tile retries configurable	2023-09-11 18:41:59 -05:00
Sean Sube	404a314050	lint(api): add typed errors from cancelled job and download error	2023-09-04 21:20:04 -05:00
Sean Sube	c506fe1e9f	make sure job is always defined, remove unused placeholders from logs	2023-07-17 22:46:02 -05:00
Sean Sube	dfe9f6ee44	fix(api): split up worker name and job name to avoid logging false failures	2023-07-15 19:00:20 -05:00
Sean Sube	7f099c02e5	reset retries with each new job	2023-07-15 18:55:38 -05:00
Sean Sube	1942c45789	apply lint	2023-07-15 16:55:52 -05:00
Sean Sube	7761aea28b	share retries throughout pipeline	2023-07-15 11:20:25 -05:00
Sean Sube	52cc4baaa4	fix circular import	2023-04-29 23:31:08 -05:00
Sean Sube	0573721d12	fix(api): always use accumulating progress counters	2023-04-29 23:28:51 -05:00
Sean Sube	30c96be24f	send token with admin requests, return worker status after restarting	2023-04-20 17:36:29 -05:00
Sean Sube	cfdd926fff	fix(api): wait for worker to become idle before enqueueing next job (#286)	2023-04-15 20:37:53 -05:00
Sean Sube	aefa5b4613	fix(api): clear job cancelled flag when worker starts a new job (#269)	2023-03-19 17:57:14 -05:00
Sean Sube	1d52dc684d	init last progress on worker context	2023-03-18 17:27:41 -05:00
Sean Sube	588c8c7fdb	fix(api): track last progress within worker	2023-03-18 15:32:49 -05:00
Sean Sube	d1565b056e	apply lint, make missing images an error	2023-03-18 15:16:41 -05:00
Sean Sube	7cf5554bef	feat(api): add error flag to image ready response	2023-03-18 15:13:42 -05:00
Sean Sube	35dc8a0bc4	improve exit logging	2023-03-05 21:37:39 -06:00
Sean Sube	c0a01efef4	fix(api): track currently active worker for each device	2023-03-05 21:28:21 -06:00
Sean Sube	3a4928e59b	fix(api): prevent workers from blocking on their progress queues	2023-03-05 20:07:06 -06:00
Sean Sube	edc55ae8b4	fix(api): finished job notification should not block worker	2023-03-05 19:53:44 -06:00
Sean Sube	21fc7c5968	fix(api): mark all convert methods as no_grad	2023-03-01 08:26:40 -06:00
Sean Sube	c99aa67220	name threads, max queues, type/lint fixes	2023-02-28 21:44:52 -06:00
Sean Sube	136759285d	set queue timeouts	2023-02-27 22:37:43 -06:00
Sean Sube	2327b24022	join all threads	2023-02-27 17:35:31 -06:00
Sean Sube	66a20e60fe	run logger in a thread, clean up status	2023-02-27 17:14:53 -06:00
Sean Sube	a37d1a4550	use progress queue	2023-02-26 20:37:22 -06:00
Sean Sube	525ee24e91	track started and finished jobs	2023-02-26 20:09:42 -06:00
Sean Sube	85118d17c6	clear worker flags between jobs, attempt to record finished jobs again	2023-02-26 15:06:40 -06:00
Sean Sube	584dddb5d6	lint all the new stuff	2023-02-26 14:15:30 -06:00
Sean Sube	e0737e9e08	update progress and finished flag from worker	2023-02-26 12:51:11 -06:00
Sean Sube	943281feb5	wire up worker jobs	2023-02-25 23:55:30 -06:00
Sean Sube	f898de8c54	background workers, logger	2023-02-25 23:49:39 -06:00