Danish striker Nikolaj Hansen headed the Icelandic home side ahead after a corner kick from former Premier League player Gylfi Sigurdsson, who later turned provider again, this time for Oliver Ekroth to double the lead.
In addition to the two set piece goals, Vikingur, who have former FC Copenhagen player Sölvi Ottesen as head coach, threatened the Brondby reaguard on numerous occasions and even managed to trible the lead towards the end against a hesitant Brondby defense.
The disappointment on Iceland was Brondby's second successive defeat after a 0-2 defeat against Viborg on Sunday. Before that Brondby had barely managed to scrape past Faroese outfit HB Torshavn, who were beaten 2-1 on aggregate.
The Danish bronze winners must improve greatly in all areas of the pitch compared to the performance on the Icelandic artificial turf if they are to proceed to a tie against French outfit Strasbourg in the playoff round before the league phase in the Conference League.
Overall the visitors' play was inconsistent even though they came close to taking the lead at the start of the game when Vikingur's Danish striker Nikolaj Hansen just managed to clear a Danish attempt off the line.
The 32-year-old striker, who has played most of his career in Iceland, was a constant threat to the Brondby defence at the other end, and just before half-time he rose to direct Sigurdsson's corner past Brondby Patrick Pentz.
History repeated itself at the start of the second half when Sigurdsson again produced an accurate corner where Swedish defender Oliver Ekroth beat everyone to the ball to double the lead.
The 35-year-old Sigurdsson then hit the post for Vikingur, while Brondby's chances at the other end were too few and far between to justify a goal.
Brondby lacked energy and belief, and those two ingredients were completely absent after 82 minutes, when a hesitant Danish defense allowed substitute Viktor Andrason to slot the ball into the net for a hugely surprising 3-0 win for the home side.