I developed xlibinput_calibrator. Here [1] you can find an implementation of the math behind the computation of the "Calibration Matrix".
Basically you need to transform the coordinates from the "touch space" to the "screen space". To do this first you need to collect at least the coordination of 3 points both in the "touch space" and "screen space". [2]
The calibration matrix (C), the coordinates in "touch space" (tx_i, ty_i, where i=1,2,3) and the coordinates in "screen space" (sx_i, sy_i) are related by the following formulas:
⎡tx₁ tx₂ tx₃⎤
⎢ ⎥
Ti = ⎢ty₁ ty₂ ty₃⎥
⎢ ⎥
⎣ 0 0 1 ⎦
⎡sx₁ sx₂ sx₃⎤
⎢ ⎥
Si = ⎢sy₁ sy₂ sy₃⎥
⎢ ⎥
⎣ 0 0 1 ⎦
⎡a b c⎤
⎢ ⎥
C = ⎢d e f⎥
⎢ ⎥
⎣0 0 1⎦
⎡a b c⎤ ⎡tx₁ tx₂ tx₃⎤ ⎡sx₁ sx₂ sx₃⎤
⎢ ⎥ ⎢ ⎥ ⎢ ⎥
⎢d e f⎥ x ⎢ty₁ ty₂ ty₃⎥ = ⎢sy₁ sy₂ sy₃⎥
⎢ ⎥ ⎢ ⎥ ⎢ ⎥
⎣0 0 1⎦ ⎣ 0 0 1 ⎦ ⎣ 0 0 1 ⎦
C * Ti = Si => C = Si * inv(Ti)
where "inv(Ti)" compute the inverse matrix of Ti. The results are (thank Octave):
sx₁⋅ty₂ - sx₂⋅ty₁
a = ─────────────────
tx₁⋅ty₂ - tx₂⋅ty₁
- sx₁⋅tx₂ + sx₂⋅tx₁
b = ───────────────────
tx₁⋅ty₂ - tx₂⋅ty₁
sx₁⋅(tx₂⋅ty₃ - tx₃⋅ty₂) + sx₂⋅(tx₁⋅ty₃ - tx₃⋅ty₁) + sx₃⋅(tx₁⋅ty₂ - tx₂⋅ty₁)
c = ──────────────────────────────────────────────────────────────────────────────
tx₁⋅ty₂ - tx₂⋅ty₁
sy₁⋅ty₂ - sy₂⋅ty₁
d = ─────────────────
tx₁⋅ty₂ - tx₂⋅ty₁
-sy₁⋅tx₂ +sy₂⋅tx₁
e = ─────────────────
tx₁⋅ty₂ - tx₂⋅ty₁
sy₁⋅(tx₂⋅ty₃ - tx₃⋅ty₂) - sy₂⋅(tx₁⋅ty₃ - tx₃⋅ty₁) + sy₃⋅(tx₁⋅ty₂ - tx₂⋅ty₁)
f = ────────────────────────────────────────────────────────────────────────────
tx₁⋅ty₂ - tx₂⋅ty₁
The one above is the "general" formula, where are considered also cases like the rotation and axis inversion (and their combination). The one that you posted is a simpler one (which need only 2 points) which may fails in these cases.
You have also to take in consideration that libinput want the matrix in the normalized form (Cn). So it becomes
⎡ b⋅sy b⋅dy⋅sy c ⎤
⎢ a ──── a⋅dx + ─────── + ── - dx⎥
⎢ sx sx sx ⎥
⎢ ⎥
⎢d⋅sx d⋅dx⋅sx f ⎥
Cn = ⎢──── e ─────── + dy⋅e - dy + ──⎥
⎢ sy sy sy⎥
⎢ ⎥
⎣ 0 0 1 ⎦
where
- sx = 1/width
- sy = 1/height
- dx = minx
- dy = miny
[1] https://github.com/kreijack/xlibinput_calibrator/blob/master/src/calibrator.cc#L144
[2] in xlibinput_calibrator, I take 4 points and the result matrix is the average of the 4 matrices computed by any combination of the points.