I read a research paper on how to train through inference by moving away from back propagation, (WAY simplifying this) by doubling the memory usage with learning neuron connections wired into the transformer layers, and having that layer and the learning layer adjusting each other doing forward inferences.
instead of doing this
0 layer
| end layer
|-->-------| tick
|----->----| tick
|-------->-| tick
|--------=>=>=>=|
|>=>=>=>=>|
|=>=>=>=>=|
|>=>=>=>=>|
|=>=>=>=>=|
|>=>=>=>=>|
the prediction and feedback mechanism is interlinked, nodes can all be processed in parallel, and multiple layers can be infering AND learning simultaneously, in waves. feedback learning is localized to the mesh grid near neighbors, and does not require a whole chain of nodes through all layers to line up and process all data together, in turn. it processes many inferences per cycle.
The 'downside' is they don't behave like fixed models we are used to any more, they learn and adapt on-the-fly. You would need to instantize, snapshot, rollback, and heavily monitor them to use them like a fixed computational tool, instead of a learning thinking entity.