Of course, the court could be "packed" as Roosevelt tried (unsuccessfully due, in part, to public outcry) to do in 1937. And, yes, that packed court could ignore the Constitution that they swore to uphold (just as a handful of people in the military could conspire to use nuclear weapons to wipe out major cities in the United States).
However, recall that 2/3 of the state legislatures can call for a Constitutional convention and 3/4 of the state legislatures can then ratify the resulting changes -- the House, the Senate, the Administration, and the Judicial branches can do nothing to stop this (well, of course, without just declaring martial law or something like that -- but, that's one reason we have the Second Amendment!). In these matters, Wyoming (the least populous state with a population of about 564,000) has the same power as California (the most populous state with a population of about 37,000,000 - or 65x the population of Wyoming). Such a convention could do almost anything except eliminate equal suffrage in the Senate (but, they could simply gut the Senate's power and add a house of Congress called NewSenate that looks a lot like the current Senate and takes on all the powers of the existing Senate but where large states have NO say!). In the extreme, states with just 40% of the population can rewrite the Constitution to undo whatever court packing scheme was enacted -- and, in theory, just 50%+1 of the people in each of those states would be needed to make this happen -- i.e., just 20% of the population in the U.S. has to agree to rewrite the Constitution. Of course, this isn't likely to happen, but it's a safety valve that discourages the kind of mischief you're proposing.
Ref:
Article. V.
The Congress, whenever two thirds of both Houses shall deem it necessary, shall propose Amendments to this Constitution, or, on the Application of the Legislatures of two thirds of the several States, shall call a Convention for proposing Amendments, which, in either Case, shall be valid to all Intents and Purposes, as Part of this Constitution, when ratified by the Legislatures of three fourths of the several States, or by Conventions in three fourths thereof, as the one or the other Mode of Ratification may be proposed by the Congress; Provided that no Amendment which may be made prior to the Year One thousand eight hundred and eight shall in any Manner affect the first and fourth Clauses in the Ninth Section of the first Article; and that no State, without its Consent, shall be deprived of its equal Suffrage in the Senate.