When rendering with Vector output, the secret to nice clean edges is 'the larger the better'.
This means making the object to be rendered in the modeling view larger than the view, if necessary. A single object that fills a fully zoomed modeling window will probably render cleanly. However a larger scene may not.
So if you are not getting nice clean edges with Vector, try expanding your modeling window as much as possible, hiding the palettes if need be. Then use "Fit Views to Selection" from the Modeling menu. If you use "Fit Views to Selection" we recommend that you hide all other objects that are not wanted in the output. You can hide objects using the Selection menu.
Or, you can use "Fit to All," and then select the Zoom Tool. Zoom in on the center of the scene. It doesn't matter that you cannot see all of the objects in the scene, the Vector render will render everything in the scene that is not hidden.
You might need to try a few renders to make adjustments, such as zooming in more closely. Also, be sure to set the "Detail Level" to High and adjust the Curve Fitting slider as needed.