The machine was a PDP-11. It was a PDP-11/45 running a one-of-a-kind graphics OS, called GRASS, the Graphics Symbiosis System written by Tom DeFanti, a professor at the University of Illinois at Chicago (then the University of Illinois at Chicago Circle). Tom's appointment then was to the Chemistry Dept.; the GRASS system was used primarily for molecular modeling. It drove an Evans & Sutherland Picture System, a giant $100,000 vector graphics engine worth five times what the PDP-11 was worth.
Larry's work pushed the system to its limits. His work was done at night, on the QT, with Tom's permission. This was done by giving Larry his own disk pack with a copy of the system on it. Larry's use of the system worked around all sorts of bugs in that relatively early version of GRASS. The film was made by pointing a (film) camera at the E&S screen, and running a macro which would render a frame, click the camera, render a frame, click the camera... While the PDP-11 system could in fact render the Death Star trench in real time, by the time you included all the little bits and frobs, the E&S took long enough to draw it that the display flickered. Hence the need to do frame-by-frame. Also, there was no frame-sync hardware in the system; the camera and display were connected only by the solenoid that tripped the camera shutter.
I played with that disk pack a year or two after the fact and it was a hoot to fly around the Death Star by hand. GRASS pioneered the interactive control of complex graphics, so all the position (and other) variables could trivially be tied to dials, etc. I was discouraged by one thing: the final version of the run had apparently been deleted from the disk. The only version I could find had the big "dish" directly on the equator of the Death Star, not at 45 degrees north latitude as in the film.
Years after that, I happened to talk to Larry Cuba by phone about something else, and asked him about that. He said the version I saw WAS the final version. Years after that, when I went to my "farewell to Star Wars viewing of Star Wars", I saw he was right. The plans shown to the rebels show the dish on the equator. Obviously the plans were fake. Those rebels were all dead men.