ag.algebraic geometry - Relative canonical sheaf

If $fcolon X to S$ a proper flat of schemes map with $n$-dimensionnal fibers over a noetherian scheme $S$, the relative canonical sheaf $omega_{X/S}:=H^n(f^!{mathcal{O}_S})$ is a dualizing sheaf. I guess that this should imply what you want by a GAGA-type argument.

Indeed, being $f$ flat we have that $f^!G = f^*G otimes f^!{mathcal{O}_S}$ [Lipman, SLN 1960, Theorem 4.9.4] for $G in D^b_c(X)$ (or, more generally, for $G in D^+_{qc}(X)$). Now, $f^!{mathcal{O}_S}$ is concentrated between degrees 0 and $-n$ by looking at the fibers of $f$ and its description via residual complexes. Then, taking $-n$-th cohomology on both sides and one obtains the desired result.

Unfortunately, I am not familiar enough with the analytic version of the story as in Ramis-Ruget-Verdier "Dualité relative en géométrie analytique complexe", but I guess the algebraic version may give you a clue for how to transpose the result to your setting. I would bet you do not need $S$ reduced as long as $f$ is flat.

Addendum

There was a confusion on my part. I was speaking about the dualizing complex while the original question was about the dualizing sheaf. If we agree that $omega_X = H^{-n}(f^!O_B)$ for a equidimensional family (with $n$ the dimension of the fibers), then the base change isomorphism in the derived category for a square with base $u colon P to B$, ($B$ for "base scheme") a base change of the map $f colon X to B$ completing the square with $v colon F to X$ and $g colon F to P$, respectively.

Then, by the formula in the derived category,

$Lv^* f^! G cong g^! Lu^* G$

we get an isomorphism of sheaves

$H^{-n}(Lv^* f^!O_B) cong H^{-n}(g^! Lu^*O_B) = H^{-n}(g'^!O_P)$

so we see that the dualizing sheaf over $F$ is the abutment of a spectral sequence involving higher $Tor$ sheaves related to the embedding of the fiber into the space $X$. But, of course one needs some information on this embedding to get the collapse of the spectral sequence.