You mean the VFX companies thought they'd chose an "incredibly slow" production renderer, rather than Arnold?
RenderMan has been a hybrid renderer for some time now
Yes I am saying that. There are a couple reasons for that. For one thing there is a lot of inertia in the industry and for good reason. You don't want to move from a tool that you know works, has worked on dozens of features previously and adopt something which might not work. Arnold is only creeping into production. Render TDs aren't familiar with raytracing in general. Lighting TDs are used to cheating everything. Only 1 production renderer historically had been the renderer of choice for feature film work (PRMan) so only it had gotten the AOVs, custom shaders, flexible scene graph manipulation and such that feature film teams want. So your choice before Arnold was essentially "We can take a mostly ready renderer like say Arnold, Brazil or Vray and hammer it into what we want or we can keep using PRMan which does everything we want albeit slowly." The "right" choice was PRMan. A few groups who didn't have the same constraints such as ILM's rogue and digimatte departments went with raytracers (for instance the opening forest shot of Avatar is a Brazil shot) also DD's commercial division was mostly Vray so when Tron came along they had a moment of power and sort of injected Vray on to Digital Domain at large (which is in my opinion why Vray suddenly started getting good focused development towards being a usable feature production renderer). And of course Arnold re-emerged after 12 years of underground development in SPI's basement. So DD and Sony did decide to put in the herculean effort to take a promising production raytracer and turn it into a competitive production renderer.
But as I'm sure you know speed isn't everything. Prman is exactly as you say all about:
... since most Blender-users are not going to be rendering multi-billion polygon scenes with massive displacement, instancing, complex shader networks, complex AOV output and custom-written Renderman shaders.
Arnold years ago: too immature to do any of that.
Brazil: massive displacement was extremely glitchy until briefly before it was acquired and killed. It also had no implicit hair spline rendering.
Vray: Had nobody using it in features so as a result it was back-asswords for feature production with feedback and bug reports only coming from Arch-Viz artists. It also didn't have the stability or capabilities of handling multi-billion poly scenes at the time
.
And none of the above had something like RIB.
So while they were all faster, if you can't rely on it for every shot it's a bad choice. You don't want to get 90% of the way into a shot and then have the renderer shit the bed and have to change renderers and redo all of your work on the shot. Renderman has always been spectacularly reliable, fast? No. Reliable, absolutely.
What finally happened though was that Arnold popped up out of stealth maturation in SPI's pipeline and made a mockery of Renderman's performance. More and more effects were raytraced and Renderman's raytracer was a kludgy tacked on piece of shit. Arnold overcame the deficiencies of Vray and Brazil for features and implemented all of the stuff that feature films want: bullet proof displacement, solid instancing, implicit hair shapes, AOVs, a RIB like scene graph with ASS plus it brought the speed and artist friendly workflow of something like Brazil or Vray.
So yes, the renderman team went back to the drawing board and turned renderman into a path tracer before they lost the entire market. But it's only been in a stable release availability for 2 years, compared to the optimization and tuning that Arnold has had going on for well over a decade. Renderman is catching up but it's definitely on defense and trying to catch up.
But all of that is irrelevant for Blender users even now you don't really want a path tracer unless you're doing animation. Most blender artists are doing essentially arch-viz and path tracing is sllllloooowwwwww. There is a reason Brazil and Arnold which were animation focused renderers spent a lot of time on their brute force GI performance while Vray which was dominating the arch-viz scene focused on the quality of its irradiance caching/Final Gather/whatever you want to call it sub-sampled GI. For stills sub-sampled GI is an order of magnitude faster. It's flickery useless garbage for animation beyond fly-throughs and such but it's what the vast majority of blender users would benefit the most from. So even with Renderman catching up the legacy raytracers on performance, it's still not focused on performance for hobbyist applications. And none of the CPU production renderers can compare to the responsiveness of a lightweight non-production renderer like Octane or Cycles for simple small scenes.