1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
|
deeptagger
==========
This is an automatic image tagger/classifier written in C++,
primarily targeting various anime models.
Unfortunately, you will still need Python 3, as well as some luck, to prepare
the models, achieved by running download.sh. You will need about 20 gigabytes
of space for this operation.
"WaifuDiffusion v1.4" models are officially distributed with ONNX model exports
that do not support symbolic batch sizes. The script attempts to fix this
by running custom exports.
You're invited to change things to suit your particular needs.
Getting it to work
------------------
To build the evaluator, install a C++ compiler, CMake, and development packages
of GraphicsMagick and ONNX Runtime.
Prebuilt ONNX Runtime can be most conveniently downloaded from
https://github.com/microsoft/onnxruntime/releases[GitHub releases].
Remember to also install CUDA packages, such as _nvidia-cudnn_ on Debian,
if you plan on using the GPU-enabled options.
$ cmake -DONNXRuntime_ROOT=/path/to/onnxruntime -B build
$ cmake --build build
$ ./download.sh
$ build/deeptagger models/deepdanbooru-v3-20211112-sgd-e28.model image.jpg
The project requires a POSIX-compatible system to build.
Options
-------
--batch 1::
This program makes use of batches by decoding and preparing multiple images
in parallel before sending them off to models.
Batching requires appropriate models.
--cpu::
Force CPU inference, which is usually extremely slow.
--debug::
Increase verbosity.
--options "CUDAExecutionProvider;device_id=0"::
Set various ONNX Runtime execution provider options.
--pipe::
Take input filenames from the standard input.
--threshold 0.1::
Output weight threshold. Needs to be set higher on ML-Danbooru models.
Tagging galleries
-----------------
The appropriate invocation depends on your machine, and the chosen model.
Unless you have a powerful machine, or use a fast model, it may take forever.
$ find "$GALLERY/images" -type l \
| build/deeptagger --pipe -b 16 -t 0.5 \
models/ml_caformer_m36_dec-5-97527.model \
| sed 's|[^\t]*/||' \
| gallery tag "$GALLERY" caformer "ML-Danbooru CAFormer"
Model benchmarks (Linux)
------------------------
These were measured with ORT 1.16.3 on a machine with GeForce RTX 4090 (24G),
and Ryzen 9 7950X3D (32 threads), on a sample of 704 images,
which took over eight hours. Times include model loading.
There is room for further performance tuning.
GPU inference
~~~~~~~~~~~~~
[cols="<,>,>", options=header]
|===
|Model|Batch size|Time
|WD v1.4 ViT v2 (batch)|16|19 s
|DeepDanbooru|16|21 s
|WD v1.4 SwinV2 v2 (batch)|16|21 s
|ML-Danbooru CAFormer dec-5-97527|16|25 s
|WD v1.4 ViT v2 (batch)|4|27 s
|WD v1.4 SwinV2 v2 (batch)|4|30 s
|DeepDanbooru|4|31 s
|ML-Danbooru TResNet-D 6-30000|16|31 s
|WD v1.4 MOAT v2 (batch)|16|31 s
|WD v1.4 ConvNeXT v2 (batch)|16|32 s
|ML-Danbooru CAFormer dec-5-97527|4|32 s
|WD v1.4 ConvNeXTV2 v2 (batch)|16|36 s
|ML-Danbooru TResNet-D 6-30000|4|39 s
|WD v1.4 ConvNeXT v2 (batch)|4|39 s
|WD v1.4 MOAT v2 (batch)|4|39 s
|WD v1.4 ConvNeXTV2 v2 (batch)|4|43 s
|WD v1.4 ViT v2|1|43 s
|WD v1.4 ViT v2 (batch)|1|43 s
|ML-Danbooru CAFormer dec-5-97527|1|52 s
|DeepDanbooru|1|53 s
|WD v1.4 MOAT v2|1|53 s
|WD v1.4 ConvNeXT v2|1|54 s
|WD v1.4 MOAT v2 (batch)|1|54 s
|WD v1.4 SwinV2 v2|1|54 s
|WD v1.4 SwinV2 v2 (batch)|1|54 s
|WD v1.4 ConvNeXT v2 (batch)|1|56 s
|WD v1.4 ConvNeXTV2 v2|1|56 s
|ML-Danbooru TResNet-D 6-30000|1|58 s
|WD v1.4 ConvNeXTV2 v2 (batch)|1|58 s
|===
CPU inference
~~~~~~~~~~~~~
[cols="<,>,>", options=header]
|===
|Model|Batch size|Time
|DeepDanbooru|16|45 s
|DeepDanbooru|4|54 s
|DeepDanbooru|1|88 s
|ML-Danbooru TResNet-D 6-30000|4|139 s
|ML-Danbooru TResNet-D 6-30000|16|162 s
|ML-Danbooru TResNet-D 6-30000|1|167 s
|WD v1.4 ConvNeXT v2|1|208 s
|WD v1.4 ConvNeXT v2 (batch)|4|226 s
|WD v1.4 ConvNeXT v2 (batch)|16|238 s
|WD v1.4 ConvNeXTV2 v2|1|245 s
|WD v1.4 ConvNeXTV2 v2 (batch)|4|268 s
|WD v1.4 ViT v2 (batch)|16|270 s
|ML-Danbooru CAFormer dec-5-97527|4|270 s
|WD v1.4 ConvNeXT v2 (batch)|1|272 s
|WD v1.4 SwinV2 v2 (batch)|4|277 s
|WD v1.4 ViT v2 (batch)|4|277 s
|WD v1.4 ConvNeXTV2 v2 (batch)|16|294 s
|WD v1.4 SwinV2 v2 (batch)|1|300 s
|WD v1.4 SwinV2 v2|1|302 s
|WD v1.4 SwinV2 v2 (batch)|16|305 s
|ML-Danbooru CAFormer dec-5-97527|16|305 s
|WD v1.4 MOAT v2 (batch)|4|307 s
|WD v1.4 ViT v2|1|308 s
|WD v1.4 ViT v2 (batch)|1|311 s
|WD v1.4 ConvNeXTV2 v2 (batch)|1|312 s
|WD v1.4 MOAT v2|1|332 s
|WD v1.4 MOAT v2 (batch)|16|335 s
|WD v1.4 MOAT v2 (batch)|1|339 s
|ML-Danbooru CAFormer dec-5-97527|1|352 s
|===
Model benchmarks (macOS)
------------------------
These were measured with ORT 1.16.3 on a MacBook Pro, M1 Pro (16GB),
macOS Ventura 13.6.2, on a sample of 179 images. Times include model loading.
There was often significant memory pressure and swapping,
which may explain some of the anomalies. CoreML often makes things worse,
and generally consumes a lot more memory than pure CPU execution.
The kernel panic was repeatable.
GPU inference
~~~~~~~~~~~~~
[cols="<2,>1,>1", options=header]
|===
|Model|Batch size|Time
|DeepDanbooru|1|24 s
|DeepDanbooru|8|31 s
|DeepDanbooru|4|33 s
|WD v1.4 SwinV2 v2 (batch)|4|71 s
|WD v1.4 SwinV2 v2 (batch)|1|76 s
|WD v1.4 ViT v2 (batch)|4|97 s
|WD v1.4 ViT v2 (batch)|8|97 s
|ML-Danbooru TResNet-D 6-30000|8|100 s
|ML-Danbooru TResNet-D 6-30000|4|101 s
|WD v1.4 ViT v2 (batch)|1|105 s
|ML-Danbooru TResNet-D 6-30000|1|125 s
|WD v1.4 ConvNeXT v2 (batch)|8|126 s
|WD v1.4 SwinV2 v2 (batch)|8|127 s
|WD v1.4 ConvNeXT v2 (batch)|4|128 s
|WD v1.4 ConvNeXTV2 v2 (batch)|8|132 s
|WD v1.4 ConvNeXTV2 v2 (batch)|4|133 s
|WD v1.4 ViT v2|1|146 s
|WD v1.4 ConvNeXT v2 (batch)|1|149 s
|WD v1.4 ConvNeXTV2 v2 (batch)|1|160 s
|WD v1.4 MOAT v2 (batch)|1|165 s
|WD v1.4 SwinV2 v2|1|166 s
|ML-Danbooru CAFormer dec-5-97527|1|263 s
|WD v1.4 ConvNeXT v2|1|273 s
|WD v1.4 MOAT v2|1|273 s
|WD v1.4 ConvNeXTV2 v2|1|340 s
|ML-Danbooru CAFormer dec-5-97527|4|445 s
|ML-Danbooru CAFormer dec-5-97527|8|1790 s
|WD v1.4 MOAT v2 (batch)|4|kernel panic
|===
CPU inference
~~~~~~~~~~~~~
[cols="<2,>1,>1", options=header]
|===
|Model|Batch size|Time
|DeepDanbooru|8|54 s
|DeepDanbooru|4|55 s
|DeepDanbooru|1|75 s
|WD v1.4 SwinV2 v2 (batch)|8|93 s
|WD v1.4 SwinV2 v2 (batch)|4|94 s
|ML-Danbooru TResNet-D 6-30000|8|97 s
|WD v1.4 SwinV2 v2 (batch)|1|98 s
|ML-Danbooru TResNet-D 6-30000|4|99 s
|WD v1.4 SwinV2 v2|1|99 s
|ML-Danbooru CAFormer dec-5-97527|4|110 s
|ML-Danbooru CAFormer dec-5-97527|8|110 s
|WD v1.4 ViT v2 (batch)|4|111 s
|WD v1.4 ViT v2 (batch)|8|111 s
|WD v1.4 ViT v2 (batch)|1|113 s
|WD v1.4 ViT v2|1|113 s
|ML-Danbooru TResNet-D 6-30000|1|118 s
|ML-Danbooru CAFormer dec-5-97527|1|122 s
|WD v1.4 ConvNeXT v2 (batch)|8|124 s
|WD v1.4 ConvNeXT v2 (batch)|4|125 s
|WD v1.4 ConvNeXTV2 v2 (batch)|8|129 s
|WD v1.4 ConvNeXT v2|1|130 s
|WD v1.4 ConvNeXTV2 v2 (batch)|4|131 s
|WD v1.4 MOAT v2 (batch)|8|134 s
|WD v1.4 ConvNeXTV2 v2|1|136 s
|WD v1.4 MOAT v2 (batch)|4|136 s
|WD v1.4 ConvNeXT v2 (batch)|1|146 s
|WD v1.4 MOAT v2 (batch)|1|156 s
|WD v1.4 MOAT v2|1|156 s
|WD v1.4 ConvNeXTV2 v2 (batch)|1|157 s
|===
Comparison with WDMassTagger
----------------------------
Using CUDA, on the same Linux computer as above, on a sample of 6352 images.
We're a bit slower, depending on the model.
Batch sizes of 16 and 32 give practically equivalent results for both.
[cols="<,>,>,>", options="header,autowidth"]
|===
|Model|WDMassTagger|deeptagger (batch)|Ratio
|wd-v1-4-convnext-tagger-v2 |1:18 |1:55 |68 %
|wd-v1-4-convnextv2-tagger-v2 |1:20 |2:10 |62 %
|wd-v1-4-moat-tagger-v2 |1:22 |1:52 |73 %
|wd-v1-4-swinv2-tagger-v2 |1:28 |1:34 |94 %
|wd-v1-4-vit-tagger-v2 |1:16 |1:22 |93 %
|===
|
