3ds Max editing collision sub shapes for custom static models in Oblivion
From Nexus Wiki
3ds Max 9 editing collision sub-shapes for custom static models in Oblivion
I've seen a lot of customized Bethesda meshes floating around, but in each and every mesh the collision meshes are left untouched. Especially complex collision meshes that include several sub shapes. Niftools plugin for 3ds Max 9 has a nasty bug that makes editing those meshes next to impossible. This tutorial is my way of getting around that bug.
3ds Max 9
Niftools Max plugin V18.104.22.16832, downloaded at Niftools sourceforge site
Use NifSkope to open the model and check how many sub shapes there are and which havok material is used. (bhkCollisionObject->...->bhkPackedNiTriStripsShape contains the list of sub shapes and their settings, including havok materials of each sub shape)
Import the model to 3dsMax. You'll notice that you can't edit the sub shaped havok meshes. This is due to bug(?) in the plugin code. To work around this, you'll need to create as many new temporary objects (boxes, for example) as there are sub shapes, minus one (the reason explained in Step 4, Note 2).
Convert those boxes to editable meshes, then add bhkRigidBody modifier and edit the settings to match the targeted sub shape properties (havok material, layer, etc.) and select Packed Strip Shape for "Bounding volume".
While having selected the havok box-shaped mesh, switch to "Top-level" selection method (in right-click menu) press "attach" button in "Edit Geometry" branch of the modifier. Select the sub-shape of the "uneditable" collision mesh. Repeat this step for each box/sub-shape(-1).
Note 1: Selecting the right sub shape is tricky and might take a lot of tries, especially if it is a small patch or if the collision mesh is complex and/or overlapping. Hiding regular objects/meshes helps. Also, while you hover the pointer over havok mesh and its sub shapes, the tool-tip will appear informing you exactly which sub shape you are keeping the pointer over. That way you can be certain that you're on the right track, if you are on the right track.
Note 2: When attaching elements, those are actually substracted from original collision mesh, thus reducing the number of sub shapes. Once the original collision mesh sub shapes number is down to 1 (not including non-mesh objects!) the mesh becomes editable, so it is not necessary to attach the last sub shape.
Locate (or select from the list) bhkRigidBody (helper object, not a modifier!) and press "add" button to add each box you created.
When done adding boxes to the collision meshes list, select each box in the view port and remove those boxes, leaving only the actual collision mesh. Repeat for each box,
Unhide all objects.
Now we're ready to export the mesh. Keep in mind what kind of mesh you're working with (if it contains animated parts, etc.) and select appropriate options. Absolute crucial is checking "collision" check-box.
A good practice is to export the mesh as a temporary mesh and use NifSkope to paste over the bhkMoppBvTreeShape branch to the "real" mesh (opened in another NifSkope) using "block/copy branch (while new bhkMoppBvTreeShape is selected, and rightclicked)" - "block/Paste At End (while nothing is selected and right-clicked on the empty space in the block list)" features (via rightclick on the node) and redirecting the "sub shape" array in bhkListShape node, or "shape" referencing link in bhkRigidBody (or bhkRigidBodyT) node to newly pasted branch by typing in the node number. Also, make sure you've deleted the old collision mesh. The simplest way is to switch the block list view to tree (View->block list view->show blocks in tree) and collapse the root node. Any nodes bellow collapsed root node should be deleted. Those aren't used in-game and can only cause CTDs alike problems. That way the regular meshes/properties/node structure are left unaltered.