Cham sought gratification and entertainment at the expense of his father's honor. He enjoyed a certain sense of superiority by witnessing Noach's shame.
After Noach and his family emerge safely from the ark, a disquieting incident occurs whereby Noach becomes intoxicated and exposes himself in his tent. The Torah records that Cham, one of Noach's three sons, "saw his father's nakedness" and brought the news of his father's humiliation to his two brothers. Shem and Yefet save their father from further embarrassment by quickly dressing him. When he comes to his senses, Noach learns of what happened and proceeds to administer a harsh curse to Cham and blessings to Shem and Yefet. Although the Torah (in 9:22) initially gives the impression that Cham merely saw his father exposed and told his brothers, which does not necessarily imply an element of disrespect, Noach's harsh response suggests that Cham did not only "see." Additionally, the Torah itself writes, "Noach awoke from his wine, and he realizes what his younger son [Cham] did to him." Clearly, Cham engaged in disrespectful behavior. Indeed, Rashi cites two views as to the objectionable and offensive conduct of Cham, actions which the Torah chose to omit, perhaps out of concern for Noach's honor.
Another possible approach to the sin of Cham would be to more carefully examine the language used to describe his reaction: "va-yaged li-shnei echav… " ("He told his two brothers"). Though a far more comprehensive study is required to verifythis thesis, the verb "le-hagid," as opposed to other verbs that mean "telling" or "reporting," connota certain emotional involvement. Perhaps the clearest example is the famous mitzva of recounting the Exodus at the seder: "You shall tell your son on that day… " ("ve-higadeta" - Shemot 13:8). As Rav Soloveitchik explains, this obligation involves far more than mere verbal recollection; it includes the recitation of hallel and active dialogue between father and son and a truly emotional experience. Later in Sefer Bereishit, when Lavan criticizes Yaakov for secretly running from town rather than bidding a warm farewell and receiving his father-in-law's blessing, Lavan complains, "ve-lo higadeta li" ("You did not tell me" - 31:27). He bemoans the fact that Yaakov never afforded him the opportunity of an emotional farewell ceremony for his children and grandchildren. In Parashat Vayechi, Yaakov prepares to administer his final charge to his sons, to tell them "that which happen to you at the end of days," and he says, "ve-agida lakhem" ("I will tell you" - 49:1).
We may reasonably speculate (particularly given the context) that herein lies the sin of Cham. He not only reported to his brothers what had happened, but he did so fervently and emphatically, relishing the rare opportunity to scoff at his aging, righteous father. (See the commentary of the Netziv, who notes the use of the term, "va-yaged" only proceeds in a different direction.)
If so, then we may perhaps better understand the curse placed upon Cham: his descendants were condemned to the service of those of Shem and Yefet. Cham sought gratification and entertainment at the expense of his father's honor. He enjoyed a certain sense of superiority by witnessing Noach's shame. He is therefore punished by having his own wishes and desires submit to the will of his masters. Whereas he had hoped to capitalize on the disadvantages of others to further his own selfish interests, he would now be compelled to serve only the interests of others, at the expense of his own dreams and aspirations.