CUDA benchmark notes#
These are the recorded experiments maintained in the repository’s benchmark notes. See GPUTreeIG for installation and the summary comparisons.
Recorded experiments#
cuda_prediction.py compares the unchanged CPU backend with both the stateless
CUDA prototype and public persistent GPUTreeIG. The CUDA path parallelizes
the complete observation x baseline x tree product and retains the CPU interval
traversal, segment ordering, and endpoint ownership policy. The stateless
measurement includes device allocation and all fixed-state transfers. The
persistent measurement keeps the model, baseline distribution, weights, and
baseline-tree cache resident, reuses device work buffers, and transfers
each observation batch, output-buffer initialization, and results. JIT compilation, model parsing, and
baseline-leaf preparation are warmed outside all timings.
GPU use remains an explicit choice through GPUTreeIG; ordinary TreeIG never
automatically dispatches to it. The local Apple-silicon development host has no
CUDA device, so it validates semantics through NUMBA_ENABLE_CUDASIM=1;
simulator timings are not performance evidence.
Each CUDA thread owns model-specialized local DFS and segment scratch. The DFS stack is bounded by maximum tree depth plus one; the segment buffer is bounded by maximum leaf count. Models requiring either power-of-two width above 1,024 entries are rejected explicitly.
Colab T4 results#
The prototype was run on a free Google Colab Tesla T4 on August 28, 2026.
The matrix used 100 trees, 12 features, weighted baselines, one warmed timing
sample per cell, and transfer-inclusive GPU timings. CPU and GPU attributions
agreed to maximum absolute error below 1.7e-14 in every cell.
Model |
n |
B |
CPU ms |
GPU ms |
Speedup |
|---|---|---|---|---|---|
boosting, depth 3 |
10 |
100 |
54.30 |
6.97 |
7.79x |
boosting, depth 3 |
100 |
10 |
69.84 |
7.56 |
9.24x |
boosting, depth 3 |
100 |
100 |
467.03 |
19.36 |
24.12x |
boosting, depth 3 |
1000 |
10 |
410.63 |
28.40 |
14.46x |
boosting, depth 3 |
1000 |
100 |
3242.77 |
140.73 |
23.04x |
forest, depth 6 |
10 |
100 |
44.26 |
12.62 |
3.51x |
forest, depth 6 |
100 |
10 |
42.61 |
12.70 |
3.35x |
forest, depth 6 |
100 |
100 |
437.08 |
97.67 |
4.47x |
forest, depth 6 |
1000 |
10 |
435.94 |
97.29 |
4.48x |
forest, depth 6 |
1000 |
100 |
5503.32 |
943.98 |
5.83x |
Small workloads remain CPU-favorable because launch, allocation, and transfer costs dominate. The shallow model demonstrates 10x-class gains once the task grid is large enough. The deeper forest’s lower ceiling supports the expected local-scratch and occupancy concern; reducing per-thread state or grouping work cooperatively is the next optimization target.
A follow-up production-style run kept the model, baseline distribution, and
baseline-tree cache resident through GPUTreeIG, reused its device buffers,
and reported the median of three warmed calls:
Model |
n |
B |
CPU ms |
Persistent GPU ms |
Speedup |
|---|---|---|---|---|---|
boosting, depth 3 |
10 |
100 |
26.00 |
2.69 |
9.68x |
boosting, depth 3 |
100 |
10 |
24.95 |
2.71 |
9.22x |
boosting, depth 3 |
100 |
100 |
252.24 |
15.22 |
16.58x |
boosting, depth 3 |
1000 |
10 |
259.69 |
15.22 |
17.06x |
boosting, depth 3 |
1000 |
100 |
2415.75 |
137.41 |
17.58x |
forest, depth 6 |
10 |
100 |
46.65 |
10.97 |
4.25x |
forest, depth 6 |
100 |
10 |
41.06 |
10.75 |
3.82x |
forest, depth 6 |
100 |
100 |
419.78 |
95.58 |
4.39x |
forest, depth 6 |
1000 |
10 |
425.98 |
95.70 |
4.45x |
forest, depth 6 |
1000 |
100 |
4447.77 |
940.57 |
4.73x |
These timings used a fresh random dataset and repeated measurements, so they
should not be compared cell-for-cell with the earlier single-sample stateless
matrix. They establish the intended deployment result: persistent execution
makes 10x-class shallow-ensemble gains available at smaller batches, while it
does not remove the deeper-tree kernel bottleneck. Maximum CPU/GPU attribution
difference was below 2.0e-14 throughout.
Deep-tree scratch specialization#
The initial kernel gave all seven local arrays the same power-of-two width
based on twice the packed node count. A full depth-6 tree therefore used width
256 for every stack and segment array. The revised kernel derives independent
bounds without changing traversal or event semantics: a DFS has at most
maximum_depth + 1 pending entries, and a leaf’s unique root path emits at
most one segment. The depth-6 specialization is consequently (8, 64) for
stack and segment widths.
On the same Colab T4, dataset, model seed, and three-repeat persistent workload used for the earlier depth-6 run, GPU latency changed as follows:
n |
B |
Before ms |
Specialized ms |
Kernel improvement |
CPU speedup |
|---|---|---|---|---|---|
10 |
100 |
10.97 |
3.74 |
2.94x |
11.17x |
100 |
10 |
10.75 |
3.53 |
3.04x |
11.82x |
100 |
100 |
95.58 |
22.63 |
4.22x |
17.02x |
1000 |
10 |
95.70 |
23.64 |
4.05x |
18.83x |
1000 |
100 |
940.57 |
207.96 |
4.52x |
19.68x |
The same implementation reached 8.76–12.84x CPU speedup for a depth-8 forest
with (16, 128) scratch widths. A depth-10 forest with (16, 256) widths
reached 7.99x at n=10, B=100, 9.00x at n=100, B=100, and 10.46x at
n=1000, B=100. Maximum absolute CPU/GPU difference across these runs was
2.62e-14.
This removes local scratch allocation as the dominant depth-6 bottleneck and meets the 10x target for large workloads through depth 10. The next likely limits are the growing leaf-segment buffer and global atomic attribution updates. Neither segment sorting nor endpoint probes were changed in this optimization.