1
0
Fork 0

fix(docs): cover extra models in user guide

This commit is contained in:
Sean Sube 2023-02-10 23:06:47 -06:00
parent 2cc604823b
commit 24c542a952
Signed by: ssube
GPG Key ID: 3EED7B957D362AF1
1 changed files with 104 additions and 21 deletions

View File

@ -22,10 +22,13 @@ Please see [the server admin guide](server-admin.md) for details on how to confi
- [Contents](#contents) - [Contents](#contents)
- [Outline](#outline) - [Outline](#outline)
- [What is ONNX web (and what it is not)](#what-is-onnx-web-and-what-it-is-not) - [What is ONNX web (and what it is not)](#what-is-onnx-web-and-what-it-is-not)
- [ONNX models](#onnx-models)
- [Modes and tabs](#modes-and-tabs) - [Modes and tabs](#modes-and-tabs)
- [Image history](#image-history) - [Image history](#image-history)
- [Scheduler comparison](#scheduler-comparison) - [Scheduler comparison](#scheduler-comparison)
- [Models](#models)
- [Adding your own models](#adding-your-own-models)
- [Model Names](#model-names)
- [Model sources](#model-sources)
- [Tabs](#tabs) - [Tabs](#tabs)
- [Txt2img tab](#txt2img-tab) - [Txt2img tab](#txt2img-tab)
- [Scheduler parameter](#scheduler-parameter) - [Scheduler parameter](#scheduler-parameter)
@ -98,26 +101,6 @@ the Gimp, Krita, or Photoshop.
This is _not_ a tool for building new ML models. While I am open to some training features, like Dreambooth and anything This is _not_ a tool for building new ML models. While I am open to some training features, like Dreambooth and anything
needed to convert models, that is not the focus and should be limited to features that support the other tabs. needed to convert models, that is not the focus and should be limited to features that support the other tabs.
### ONNX models
The [ONNX runtime](https://onnxruntime.ai/) is a library for accelerating neural networks and machine learning models,
using [the ONNX file format](https://onnx.ai/) to share them across different platforms. ONNX web is a server to run
hardware-accelerated inference using those models and a web client to provide the parameters and view the results.
The models used by ONNX web are split up into three groups:
1. Diffusion
1. general models like [Stable Diffusion](https://huggingface.co/runwayml/stable-diffusion-v1-5)
2. specialized models like [Knollingcase](https://huggingface.co/Aybeeceedee/knollingcase) or [OpenJourney](https://huggingface.co/prompthero/openjourney)
2. Upscaling
1. [Real ESRGAN](https://github.com/xinntao/Real-ESRGAN)
2. [Stable Diffusion](https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler)
3. Correction
1. [CodeFormer](https://github.com/sczhou/CodeFormer)
2. [GFPGAN](https://github.com/TencentARC/GFPGAN)
There are many other models available and specialized variations for anime, TV shows, and all sorts of other styles.
### Modes and tabs ### Modes and tabs
- [txt2img](#txt2img-tab) - [txt2img](#txt2img-tab)
@ -150,6 +133,106 @@ image from history if you don't like it.
![tilt shift photographs of excavators over a pile of loose dirt](output/excavator-pipe.png) ![tilt shift photographs of excavators over a pile of loose dirt](output/excavator-pipe.png)
## Models
The [ONNX runtime](https://onnxruntime.ai/) is a library for accelerating neural networks and machine learning models,
using [the ONNX file format](https://onnx.ai/) to share them across different platforms. ONNX web is a server to run
hardware-accelerated inference using those models and a web client to provide the parameters and view the results.
The models used by ONNX web are split up into three groups:
1. Diffusion
1. general models like [Stable Diffusion](https://huggingface.co/runwayml/stable-diffusion-v1-5)
2. specialized models like [Knollingcase](https://huggingface.co/Aybeeceedee/knollingcase) or [OpenJourney](https://huggingface.co/prompthero/openjourney)
2. Upscaling
1. [Real ESRGAN](https://github.com/xinntao/Real-ESRGAN)
2. [Stable Diffusion](https://huggingface.co/stabilityai/stable-diffusion-x4-upscaler)
3. Correction
1. [CodeFormer](https://github.com/sczhou/CodeFormer)
2. [GFPGAN](https://github.com/TencentARC/GFPGAN)
There are many other models available and specialized variations for anime, TV shows, and all sorts of other styles.
### Adding your own models
You can convert and use your own models without making any code changes by adding them to a JSON or YAML file using
the extras schema:
```json
{
"diffusion": [
{
"name": "diffusion-knollingcase",
"source": "Aybeeceedee/knollingcase"
},
{
"name": "diffusion-openjourney",
"source": "prompthero/openjourney"
},
{
"name": "diffusion-stablydiffused-aesthetic-v2-6",
"source": "civitai://6266?type=Pruned%20Model&format=SafeTensor",
"format": "safetensors"
},
{
"name": "diffusion-unstable-ink-dream-onnx-v6",
"source": "civitai://5796",
"format": "safetensors"
}
],
"correction": [],
"upscaling": [
{
"name": "upscaling-real-esrgan-x4-anime",
"source": "https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth",
"scale": 4
}
]
}
```
Set the `ONNX_WEB_EXTRA_MODELS` environment variable to the path to your file. For example:
```shell
# on Linux:
> export ONNX_WEB_EXTRA_MODELS=~/onnx-web-extras.json
# on Windows:
> set ONNX_WEB_EXTRA_MODELS=C:\Users\ssube\onnx-web-extras.json
```
Files using the older extras format with nested arrays (`"diffusion": [[]]`) will be loaded in the newer format. You
only need to convert them into the newer format if you need to use keys other than `name`, `source`, and `scale`.
#### Model Names
The `name` of each model dictates which category it will appear in on the client.
- `diffusion-*` or `stable-diffusion-*` for diffusion models
- `upscaling-*` for upscaling models
- `correction-*` for correction models
Models that do not match one of the prefixes will not be shown, so if you cannot find a model that you have converted,
make sure it is named correctly.
#### Model sources
You can provide an absolute or relative path to a local model, and there are a few pre-defined sources from which models can be downloaded.
- `huggingface://`
- https://huggingface.co/models
- mostly SFW
- requires an account for some
- `civitai://`
- https://civitai.com/
- some NSFW
- does not require an account
- `https://`
- any other HTTPS source
If the model's `source` does not include a file extension like `.safetensors` or `.ckpt`, make sure to indicate the
file format using the `format` key.
## Tabs ## Tabs
### Txt2img tab ### Txt2img tab