I  The Catholic teaching on this matter is given to us by God Himself in the Bible:

Mark X 11-12:  "And He saith to them: Whosoever shall put away his wife and marry another, committeth adultery against her.  And if the wife shall put away her husband, and be married to another, she committeth adultery."

Luke XVI 18:  "Every one that putteth away his wife, and marrieth another, committeth adultery: and he that marrieth her that is put away from her husband, committeth adultery."

Rom. VII 2:  "For the woman that hath an husband, whilst her husband liveth is bound by the law.  But if her husband be dead, she is loosed from the law of her husband."

ROM VII 3:  "Therefore, whilst her husband liveth, she shall be called an adulteress, if she be with another man: but if her husband be dead, she is delivered from the law of her husband; so that she is not an adulteress, if she be with another man."

I Cor. VI, 9-10:  "Know you not that the unjust shall not possess the kingdom of God?  Do not err: Neither fornicators, nor idolaters, nor adulterers, nor the effeminate, nor liars with mankind, (...) shall possess the kingdom of God."

1 Cor. VII 10-11:  "But to them that are married, not I, but the Lord, commandeth that the wife depart not from her husband.  And if she depart, that she remain unmarried, or be reconciled to her husband.  And let not the husband put away his wife."

1 Cor. VII 39:  "A woman is bound by the law as long as her husband liveth; but if her husband die, she is at liberty.  Let her marry to whom she will, only in the Lord."

This is why adultery is a "sinful union," (1) preventing those who commit it from going to heaven (2).  Moreover, they must "be excluded from the Sacraments" (3), except confession which they can make provided they have the proper dispositions.

II  Why is adultery a grievous sin, excluding from the Heavenly Kingdom?

A  Because adultery is against chastity.
In adultery, a married person has sexual relations with another, married or not, who is not his/her spouse. Only marriage authorizes this relationship between two persons of different sex.  Thus adultery violates chastity which is obligatory outside marriage.

B  Because adultery is an injustice to the faithful spouse.
By adultery, a married person has sexual relations with another, married or not, who is not his/her spouse but through marriage, the spouses mutually give themselves to each other, exclusively and perpetually.  Thus adultery is unjust toward the faithful spouse because it gives to another that which he (or she) alone has the exclusive right to receive.

C  Because adultery is an injustice to the legitimate children.
Through adultery, one risks having illegitimate children or provoking a divorce, if the adultery becomes known.  The existence of illegitimate children or the possibility of a divorce jeopardizes the care that the parents should give to their legitimate children.  Thus, adultery is unjust to the legitimate children because it violates their well-being.

III  Question:  If life between the spouses becomes intolerable, may not one look for a little affection or pleasure with a third person?

IV  Answer:  "When indeed, matters have come to such a pitch that it seems impossible for them to live together any longer, then the Church allows them to live apart and strives, at the same time, to soften the evils of this separation by such remedies and helps as are suited to their condition; yet she never ceases to endeavour to bring about a reconciliation, nor despairs of doing so.  But these are extreme cases; and they would seldom exist if men and women entered into the married state with proper dispositions, not influenced by passion but entertaining ideas of the duties of marriage and of its noble purpose; neither would they anticipate their marriage by a series of sins drawing down upon them the wrath of God." (Leo XIII, Encycl. Arcanum Divinae Sapientiae, Feb. 10. 1880, M 194.)

V  Question:  If one of the spouses gives permission to the other to practice adultery does not that remove the injustice?

VI  Answer:  In that case, such an action remains objectively a grave offense against the divine precepts which man can not change: a sin against chastity and an injustice towards the legitimate children.  This is why Pope Innocent XI solemnly declared that, in spite of the consent of the faithful spouse, such an action was always adultery (Decree of the Sacred Office of the 2nd March, 1679; Proposition 50; Ds 2150).

VII  Conclusion:
"Conjugal fidelity is attacked (...) by those who advocate modern ideas and outlook concerning a false and far from harmless friendship with third parties.  Husband and wife, they claim, should be allowed greater freedom of feeling and action in their relations with others, especially because many are naturally endowed with a sexual temperament which cannot find satisfaction within the narrow sphere of monogamous marriage.  The rigid attitude of good married folk, which condemns and banishes any sensual feeling or action in regard to a third person, is accounted by them an old-fashioned narrowness of mind, or a contemptible form of jealousy.  They therefore consider that any penal laws of the State for the safeguarding of marital fidelity are obsolete, or ought to be annulled.

"The decent feeling of every chaste husband and wife needs only to listen to the voice of nature to repudiate these opinions as both groundless and shameful; and the voice of nature is corroborated by God's commandment: ‘Thou shalt not commit adultery’ (4) and by the words of Christ, ‘Whosoever shall look upon a woman to lust after her, hath already committed adultery with her in his heart’ (5).  No human customs, no bad examples, no vaunted human progress, can deprive this divine commandment of its binding force.  Just as ‘Jesus Christ is one, yesterday and today and the same forever’ (6) so the teaching of Christ remains forever one and the same, and not one jot or tittle of it shall pass away until all is fulfilled (7)"  (Pius XI: Encyclical Casti Connubii, Dec. 31, 1930; M333 - 334).
 

For easy reference the letter M stands for "Matrimony - Papal Teachings" by the Monks of Solesmes, The Daughters of St. Paul Edition, 1963.