The & symbol in Rust is not a "dereferenced pointer." &a + &b is not pointer arithmetic. &a means that you use "a" by read-only reference (without copying it or modifying it). Therefore, it makes perfect sense to use it for arithmetic, even though it looks a bit strange. The operation will use the existing values without copying them or modifying them, and produce a new value. There is a complication, that the result of the computation is a value, not a reference, so to cover all situations you need to define the arithmetic operators for all combinations of reference + value, i.e. 4 overloads for each operation. Luckily, you can create Rust macros to generate the extra 3 overloads from the first one. You don't need to use "&" for primitive types like f64, because such types can be implicitly copied as needed. Rust defines the arithmetic operators for both f64 and &f64. I think this is the right approach for all types. Once the intermediate value is used in another operation (function call), it is "owned" by the new operation, which means that it is automatically deallocated when the operation returns, unless the operator returns it, in which case its ownership returns back to the caller. The result of an arithmetic expression is a single value, with all intermediate values cleaned-up (deallocated).
Whether or not there are pointers involved is irrelevant. With Rust, I think in terms of references, not in terms of pointers, unlike languages like C++, Java and Go which use pointers much more extensively. A pointer in Rust is usually hidden in a standard library type, like Vec or Box.
Rust has built on the experience provided by 20 years of C++, Java, and other languages. Statements like "Rust is still just a toy" and "entirely not proven" dismiss all this experience. Every good language should be a toy that programmers like playing with. You learn it by playing with it. The Rust compiler does a lot more work than other compilers, even giving suggestions about how to fix syntax/usage errors, so it becomes your tutor.