The "original" engine is kind of a hard thing to really quantify. As far as I recall the game went through a number of heavy rewrites and things of that nature over it's development time before going over to HH and becoming the Doom 3 engine game.
For those who need a refresher, or do not remember what Prey used to be like;
Prey (1998) Trailer
http://www.youtube.com/watch?v=XSFxiuID1TY
Prey (1998) Tech Demonstration
http://www.youtube.com/watch?v=L29DSSQT3Ek
It truly is sad, as I understand it the game was actually in pretty good shape.
If I were to throw out a theory as to what happened. It was probably the same thing that happened to DNF. Something came along and impressed/shocked the guys at 3DR and they decided to start over/abandon the project for fear of not being able to match up. I also seem to remember reading about all manner of problems with not having enough staff, and just some nasty bugs.
It'd be great if we could get some of the old guys in here to comment on it. That was a whole other era of gaming, and i'd love to know more about what went on in the office back in this days.
As far as what people could do with it? Honestly, with engines like Darkplaces or some of the Quake 2 source projects, an engine like the one for Prey is basically long since outdated. It wouldn't give us anything really new to work with I don't think. Especially if it wasn't working 100% as it should have been.