Can I measure the moon's gravity?

Running the math for a 5 meter long pendulum and 1 kg mass, I get an amplitude of 0,017 mm.

You are off by quite a bit. There is essentially no horizontal deflection when the Moon is at the horizon. The maximum horizontal deflection occurs when the Moon is about 45 degrees above or below the horizon.

The tidal acceleration at some point on the surface of the Earth due to some external body (e.g., the Moon) is the difference between the gravitational acceleration toward that external body at the point in question and the gravitational acceleration of the Earth as a whole toward that body:

$$vec a_{rel} = GM_text{body} left(frac {vec R - vec r} {||vec R - vec r||^3} - frac {vec R} {||vec R||^3} right)$$
where

• $M_{body}$ is the mass of the external body,


• $vec R$ is the displacement vector from the center of the Earth to the center of the external body, and


• $vec r$ is the displacement vector from the center of the Earth to the point in question on the surface of the Earth.

The resultant force looks like this:

Note that the tidal force is away from the center of the Earth when the Moon is directly overhead or directly underfoot, toward the center of the Earth when the Moon is on the horizon, and horizontal when the Moon is halfway between directly overhead/underfoot and on the horizon.

The tidal force is maximum when the Moon is directly overhead, and even then it's only about 10^-7g. You need a sensitive instrument to read that. A simple pendulum or a simple spring will not do the trick.