set theory - Mostowski collapses and universal extensional relational classes

The answer to your first question is no, unfortunately, it
is not functorial.

Here is a comparatively simple counterexample. The idea is
that two objects in $A$ can collapse to the same point, but
gain new elements in $A'$ that differentiate them. Let
$A={a,b}$ have two objects, with $R$ the empty
relation. So the Mostowski collapse of $(A,R)$ maps both
$a$ and $b$ to the emptyset $varnothing$. Let
$A'={a,b,c}$, with $cmathrel{R'}b$, but otherwise $R'$
is trivial. So the Mostowski collapse of $(A',R')$ maps
$a,cmapstovarnothing$ and $bmapsto{varnothing}$.
Let $f:Ato A'$ be the inclusion map, which is a
homomorphism, since $a$ and $b$ have no relation with
respect to $R$ or $R'$. Since we have $G(a)= G(b)$ but
$G'(f(a))neq G'(f(b))$, there can be no commutative
diagram here.

But why not insistin on extensionality? In the category of extensional class relations, that is, if your structures $(A,R)$ and $(A',R')$ satisfy
extensionality, then the Mostowski collapse is functorial. This is because $G$
and $G'$ are isomorphisms, and it is easy to see that there
is the induced homomorphism $h:Mto M'$ defined by
$h(G(a))=G'(f(a))$. This is a homomorphism since $G(a)in
G(b)iff amathrel{R} biff f(a)mathrel{R'}f(b)iff
G'(f(a))in G'(f(b))$, and the diagram commutes by
definition.

For the second question, in this extensional case, since
$(M',{in})$ is isomorphic to $(A',R')$, then we get a map
$(M,{in})to (A',R')$, as you say. It is just the map
$hcirc G^{-1}$. In general, there is not a unique map from
$(M,{in})$ to $(A',R')$, however, since for example, when
$(A,R)$ is a well-order, then the Mostowski collapse is an
ordinal, but if $(A',R')$ is a longer ordinal, there could
be many homomorphisms of $(M,{in})$ to $(A',R')$, since
these are just the order-preserving maps. But this map will
be unique such that the diagram commutes, since it is
determined by those isomorphisms. (But I think you were not
actually asking this about the extensional case. I'm not
sure what you mean by "adding transitivity", since the
Mostowski collapse of any structure is a transitive set,
even when the relations are not extensional.)

If you really don't want extensionality, then it would also be sensible to restrict the homomorphism concept to functions $f:Ato A'$ which not only preserve the relation (in both directions), but which also respect equivalence of predecessors; this amounts to being well-defined on the Mostowski collapse. That is, in this category, we think of $langle A,Rrangle$ as a code for its Mostowski collapse, and the notion of homomorphism should respect that. In this case, the Mostowski collapse will again be functorial.

Lastly, let me mention that several other restrictions of your homomorphism concept are often studied. Namely, in set theory the concept of a $Sigma_n$-elementary map is prominent for natural numbers $n$. Even $Sigma_0$-elementary goes beyond the basic concept of homomorphism that I think you intend, but we are often interested in $Sigma_1$-elementary embeddings or even fully elementary embeddings. Many large cardinal concepts, for example, are characterized by the existence of such embeddings defined on the entire universe into a transitive class.