Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 



Forgot your password?
typodupeerror

Comment Re: Finally (Score 1) 23

Is "Attention is all you need" weird?

"We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. "

"In this work we propose the Transformer, a model architecture eschewing recurrence and instead relying entirely on an attention mechanism to draw global dependencies between input and output. The Transformer allows for significantly more parallelization and can reach a new state of the art in translation quality"

"the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution."

"An attention function can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. The output is computed as a weighted sum of the values, where the weight assigned to each value is computed by a compatibility function of the query with the corresponding key."

"We call our particular attention "Scaled Dot-Product Attention" (Figure 2). The input consists of queries and keys of dimension
d
k
, and values of dimension
d
v
. We compute the dot products of the query with all keys, divide each by
d
k
, and apply a softmax function to obtain the weights on the values."

Can you please explain how attention is a neural network and not a simple dot product calculation requiring no neurons or hidden layers?

Comment Re: Too many old people (Score 1) 116

"letting them take those benefits away from their children and grandchildren..."

What if you don't have children?

Also, for breeders, why have I been hearing this argument since Reagan, but the debt has gone up over 500% and King Dollar means we still have an exorbitant privilege to print money (sell new bonds faster than old ones are redeemedl so those grandchildren don't have to pay anything back and creditors all remain whole and profitable?

Tl;dr: can you stop with the "taking from your grandchildren" meme, because it's been proven wrong already?

Comment Re: Finally (Score 1) 23

" AI models have "gotten really good over the last 10 years at being able to pull those types of signals out of noise," Hoiland said..."

How much does the attention mechanism (which is not a neural network, and which was a paradigm shift in chatbot proficiency at natural language) play a part in any pattern finding?

Comment Re:Makes no sense (Score 1) 59

The spec is not for competing implementations. The spec is for assuring properties of code. Something critically important in secure coding. Yes, many people do not understand that, but most people do not understand what secure coding actually means. Yes, the implementation may still have bugs, but with a spec you can reliably find out whether something is a bug or not.

Comment Re: Energiewende (Score 1) 117

Did I claim renewables were "CO2 free"? No, I did not. And they are not.

The point is: the intrinsic method of energy generation is CO2 free. The rest is dependent on the "state of technology".

What a nice lie by misdirection you have there. Protip: By that measure, _all_ energy generation forms can be made CO2 free. You are just pushing nonsense.
In the real world, the one which you do not live in, it matters how much CO2 gets released, no matter why, per amount of energy generated.

Slashdot Top Deals

To stay youthful, stay useful.

Working...