I went to the second week of the Los Angeles Olympics in 1984. My main advice is: only attend the events at which gold medals are handed out. Stay away from preliminary rounds.
The excitement of a once every four years prize makes minor sports fun for a day. For example, a weightlifting final I attended turned out to be a blast, almost as much fun as professional wrestling (the weightlifters try to psych each other out, which is fun to watch). This was their moment in the spotlight and they took full advantage of it. In contrast, watching the American basketball team starring Michael Jordan and Patrick Ewing win a fairly close basketball quarterfinal match over West Germany 78-67 sounds pretty good in theory, but was actually rather dull.