Do you have an oscilloscope ?
Although unlikely it might be a problem with the opamp oscillating, some opamps are not "unity-gain stable".
I expect the LMV324 to be unity-gain stable: the old National semiconductor datasheet says this:
The LMV321/358/324 can directly drive 200pF in unity-gain without oscillation.
If you have an oscilloscope it might not hurt to check.
Another check: do you have the unused inputs of the opamp connected to a fixed voltage ? Keeping unused inputs floating can also cause erratic behavior of the opamp. No other outputs connected to a lower impedance to ground than 3Kohms ?