This is a philosophical question, rather than a matehmatical one. Anyway personally (it's a metter of personal taste!) I totally agree that mathematics is more about correctness than about truth.
In the following paragraphs I will try to (partially) answer your specific doubts about Goedel incompleteness in a down to earth way, with the caveat that I'm no expert in logic nor I am a philosopher. (See also this MO question, from which I will borrow a piece of notation). I had some doubts about whether to post this answer, as it resulted being a bit too verbose, but in the end I thought it may help to clarify the related philosophical questions to a non-mathematician, and also to myself.
The point is that there are several "levels" in which you can "state" a certain mathematical statement; more: in theory, in order to make clear what you formally want to state, along with the informal "verbal" mathematical statement itself (such as $2+2=4$) you should specify in which "level" it sits. (Of course, as mathematicians don't want to get crazy, in everyday practice all of this is left completely as understood, even in mathematical logic).
For example, suppose we work in the framework of Zermelo-Frenkel set theory ZF (plus a formal logical deduction system, such as Hilbert-Frege HF): let's call it Set1. In this setting, you can talk formally about sets and draw correct (relative to the deduction system) inferences about sets from the axioms. You can also formally talk and prove things about other mathematical entities (such as $mathbb{N}$, $mathbb{R}$, algebraic varieties or operators on Hilbert spaces), but everything always boils down to sets.
Still in this framework (that we called Set1) you can also play the game that logicians play: talking, and proving things, about theories $T$. How? Well, you only have sets, and in terms of sets alone you can define "logical symbols", the "language" $L$ of the theory you want to talk about, the "well formed formulae" in $L$, and also the set of "axioms" of your theory. Examples of such theories are Peano arithmetic PA (that in this incarnation we should perhaps call PA2), group theory, and (which is the reason of your perplexity) a version of Zermelo-Frenkel set theory ZF as well (that we will call Set2). Note that every piece of Set2 "is" a set of Set1: even the "$in$" symbol, or the "$=$" symbol, of Set2 is itself a set (e.g. a string of 0's and 1's specifying it's ascii character code...) of which we can formally talk within Set1, likewise every logical formula regardless of its "truth" or even well-formedness. Stating that a certain formula can be deduced from the axioms in Set2 reduces to a certain "combinatorial" (syntactical) assertion in Set1 about sets that describe sentences of Set2.
The good think about having a meta-theory Set1 in which to construct (or from which to see) other formal theories $T$ is that you can compare different theories, and the good thing of this meta-theory being a set theory is that you can talk of models of these theories: you have a notion of semantics.
In the light of what we've said so far, you can think of the statement "$2+2=4$" either as a statement about natural numbers (elements of $mathbb{N}$, constructed as "finite von Neumann ordinals" within Set1, for which $0:=emptyset$, $1:=${$emptyset$} etc.); or as a sentence of PA2 (which is actually itself a bare set, of which Set1 can talk).
An interesting (or quite obvious?) thing is that in some cases it makes sense to go on to "construct theories" also within the lower levels. For example, within Set2 you can easily mimick what you did at the above level and have formal theories, such as ZF set theory itself, again (which we can call Set3)!
A crucial observation of Goedel's is that you can construct a version of Peano arithmetic not only within Set2 but even within PA2 itself (not surprisingly we'll call such a theory PA3). So you have natural numbers (of which PA2 formulae talk of) codifying sentences of Peano arithmetic!
So, if we loosely write "$A-triangleright B$" to indicate that the theory or structure $B$ can be "constructed" (or "formalized") within the theory $A$, we have a picture like this:
Set1 $-triangleright$ ($mathbb{N}$; PA2 $-triangleright$ PA3; Set2 $-triangleright$ Set3; T2 $-triangleright$ T3; ...).
So, you see that in some cases a theory can "talk about itself": PA2 talks about sentences of PA3 (as they are just natural numbers!), and there is a formally precise way of stating and proving, within Set1, that "PA3 is essentially the same thing as PA2 in disguise".
Furthermore, you can make sense of otherwise loose questions such as "Can the theory $T$ prove it's own consistency?". How? Well, you construct (within Set1) a version of $T$, say T2, and within T2 formalize another theory T3 that also "works exatly as $T$". Then you have to formalize the notion of proof.
So, the Goedel incompleteness result stating that
"Peano arithmetic cannot prove its own consistency"
is really a theorem of Set1 asserting that "PA2 cannot prove the consistency of PA3". This means: however you've codified the axioms and formulae of PA as natural numbers and the deduction rules as sentences about natural numbers (all within PA2), there is no way, manipulating correctly the formulae of PA2, to obtain a formula (expressed of course in terms of logical relations between natural numbers, according to your codification) that reads like "It is not true that axioms of PA3 imply $1neq 1$".
You can say an exactly analogous thing about Set2 $-triangleright$ Set3, and likewise about every theory "at least compliceted as PA".
Now, about truth. First of all, the distinction between provability a and truth, as far as I understand it. It is easy to say what being "provable" means for a formula in a formal theory $T$: it means that you can obtain it applying correct inferences starting from the axioms of $T$. This is a purely syntactical notion.
The concept of "truth", as understood in the semantic sense, poses some problems, as it depends on a set-theory-like meta-theory within which you are supposed to work (say, Set1). Saying that a certain formula of $T$ is true means that it holds true once interpreted in every model of $T$ (Of course for this definition to be of any use, $T$ must have models!).
If we could convince ourselves in a rigorous way that ZF was a consistent theory (and hence had "models"), it would be great because then we could simply define a sentence to be "true" if it holds in every model. This was Hilbert's program. Unfortunately, as said above, it is impossible to rigorously (within ZF itself for example) prove the consistency of ZF.
About true undecidable statements.
The assertion of Goedel's that
"There is a property of natural numbers that is true but unprovable from the axioms of Peano arithmetic"
is a theorem of Set1 stating that there is a sentence of PA2 that holds true* in any model of PA2 (such as $mathbb{N}$) but is not obtainable as the conclusion of a finite set of correct logical inference steps from the axioms of PA2.
*(that a sentence of PA2 is "true in any model" here means: "the corresponding interpretation of that sentence in each model, which is a sentence of Set1, is a consequence of the axioms of Set1")
According to Goedel's theorems, you can find undecidable statements in any consistent theory which is rich enough to describe elementary arithmetic. That is, such a theory is either inconsistent or incomplete.
About meaning of "truth".
Foundational problems about the absolute meaning of truth arise in the "zeroth" level, i.e. about sentences expressed in what is supposed to be the foundational theory Th0 for all of mathematics According to some, this Th0 ought to be itself a formal theory, such as ZF or some theory of classes or something weaker or different; and according to others it cannot be prescribed but in an informal way and reflect some ontological -or psychological- entity such as the "real universe of sets".
I would roughly classify the former viewpoint as "formalism" and the second as "platonism".
One point in favour of the platonism is that you have an absolute concept of truth in mathematics. One drawback is that you have to commit an act of faith about the existence of some "true universe of sets" on which you have no rigorous control (and hence the absolute concept of truth is not formally well defined). According to platonism, the Goedel incompleteness results say that
"Logic cannot capture all of mathematical truth".
On the other hand, one point in favour of "formalism" (in my sense) is that you don't need any ontological commitment about mathematics, but you still have a perfectly rigorous -though relative- control of your statements via checking the correctness of their derivation from some set of axioms (axioms that vary according to what you want to do). One consequence (not necessarily a drawback in my opinion) is that the Goedel incompleteness results assume the meaning:
"There is no place for an absolute concept of truth: you must accept that mathematics (unlike the natural sciences) is more a science about correctness than a science about truth".