Have you checked the vehicle manual? If it doesn't prohibit installing a car seat in that location, try it out. Often, after you get the child seat installed firmly in front of the arm rest, it is held in place and can no longer be pulled down. There are no documented cases of injury due to a fold down arm rest interacting with a rear facing safety seat.
If you do find that the instruction manual prohibits installing a safety seat in that position or if the arm rest still flops down freely after you have the seat properly installed* in that position, then you'll have to use an outboard position. Statistically, both sides are virtually identical when it comes to side impacts, so you'll just have to look at which side is more convenient. If you frequenly parallel park on busy streets, you'll probably want to install it on the curb-side (usually passenger side, unless it's a one-way street like I live on). If you don't, you'll probably find the driver's side is more convenient, because that's the side the driver is on anyway. Sometimes a rear facing seat will take up extra space so the seat in front of it has to be moved forward. You'll need to leave at least 10" between the driver/passenger and a frontal airbag, so that's something else to consider.
*A properly installed seat will move no more than 1" from side to side or from front to back when gripped at the belt path and tugged firmly. (You're trying to move the seat, not rip it from the vehicle.) Most rear facing seats will "flip" toward the back of the car if you push up on the top of the seat. That's
called "cocooning", and it's normal. A
certified Child Passenger Safety Technician can assist you if you have difficulty getting a good install in your vehicle.