Are you interested in becoming a particle theorist or experimentalist?
A really nice book on particle physics, is
"Introduction to Elementary Particles"
by David Griffiths
It's one of the standard textbooks used for particle physics courses over the last 20 years. It is also one of the more readable and "painless" ones too. (Most books on particle physics have a reputation for being difficult to read). I wish I first read this book when I was younger.
The first two chapters of Griffiths can be read without much extensive background. It's mainly about the history and background of particle physics, using only simple algebra. Do most of the end-of-chapter problems from the first two chapters. (They are fairly simple). If these simple problems keep your interest and/or whet your appetite for more, then it may be worthwhile to learn some special relativity on your own.
The third chapter in Griffiths is a review of special relativity, which is the standard bread-and-butter of particle physics. Griffiths assumes the reader already knows some basic special relativity. Most of the processes studied in particle physics experiments are relativistic collisions, which Griffiths' third chapter is focussed on.
Easier sources for learning special relativity, would be to go to a generic freshman university physics textbook and work out some of the theory and end-of-chapter problems. For example, work out for yourself and understand how E = m c^2 is derived, as well as some of the problems for simple collision for both the relativistic + non-relativistic cases and understand how they differ from one another. (The generic non-relativistic collision problems are frequently based on things like pool balls hitting one another, automobiles crashing into one another, etc
...).
The rest of Griffiths may look kind of cryptic and mysterious, especially the last chapter on gauge theories. If you plan on becoming a particle theorist, you will certainly be studying gauge theories in the context of quantum field theory quite extensively. Most major universities have a year long sequence of quantum field theory courses, which most particle theory folks will take at the masters degree or PhD level. (Quantum field theory is not usually covered at the undergraduate level).
http://en.wikipedia.org/wiki/Quantum_field_theoryTo really appreciate the rest of Griffiths' particle book, you will have to know some basic quantum mechanics. It turns out, Griffiths also wrote a nice quantum mechanics textbook which is also quite popular and widely used for undergraduate physics courses on quantum mechanics. (Many quantum mechanics textbooks have a reputation for being difficult to read). Quantum mechanics will generally require some understanding of calculus and simple differential equations.
If you're really "die hard" about particle physics, it may be worthwhile to purchase both the elementary particles and quantum mechanics textbooks by Griffiths. (It may be worthwhile to find second handed copies of the first editions of each book. Later editions are easily over a hundred dollars each for a brand new copies). It will give you something to think about and various calculations to figure out yourself. They are the two easiest books on the subject, without being dumbed down.
http://en.wikipedia.org/wiki/David_Griffiths_(physicist)These two Griffiths books (elementary particles, and quantum mechanics) will probably determine whether particle physics will keep your interest or not. Hopefully it will keep your interest, despite physics being a hard subject.