ds.dynamical systems - What is the definition of ideal boundary?

There are infinitely (uncountably:-) many definitions of the ideal boundary.
So you have to specify which paper you are reading:-)

See, for example M. Brelot, On topologies and boundaries in potential theory
and Constantinescu, Cornea, Ideale Ränder Riemannscher Flächen.

The simplest example of an ideal boundary is one point compification.
But depending on context, one usually has a space of functions, and one wants
to add ideal bundary points so that the functions of your space have limits.
(Examples: Prime ends, Martin boundary, etc.)