well, the 1980 Games were boycotted by 64 Nations (65 if we count Liberia, too...they participated to the Opening Ceremony but then withdrew from all the actual sport events)...
among them, USA of course, but also Canada, West Germany, Japan, Kenya (just to name those who had more medal chances than the others)...even Iran and many othe Middle East Nations...
Countries like Italy, France and Great Britain, instead, decided to participate without their athletes enrolled in the military forces and under the Olympic Flag...
the 2 most prominent Italian names missing the Olympics despite being huge medal candidates were Mariano Scartezzini (that year he was the 2nd best 3000m steeple chase specialist behind a Kenian, who was also absent) and Felice Mariani (judo bronze medallist in 1976 and among the favourites in the -60kg in that year)...
not sure about some fencing specialist, though...
for what concerns the US team, instead, here you can find the full swimming and athletics teams, with many of those athletes that could easily be considered "miss medals" because of the boycott (the 2 biggest names to me are Mary T. Meagher and Tracey Caulkins in swimming and Edwin Moses and Renaldo Nehemiah in athletics, of course)...
Swimming Team 1980...http://magazines.swimmingworld.com:9997/spipdf/1980usolympictrials.pdf
Athletics Trials 1980...https://en.wikipedia.org/wiki/1980_United_States_Olympic_Trials_(track_and_field)
p.s. in that T&F team, there should have been also Matt Centrowicz Sr., dad of Matt Centrowicz Jr., who won the 1500m in Rio...