Did they think that Allah (swt) would not punish them when they prevented the Prophet (saws) and the believers from praying and circumambulating in the Sacred House and when they violated the terms of the Hudaybiyyah treaty, prohibiting the believers from performing Umrah? They are not the rightful custodians of the Sacred House; the true ones are those       who follow the path of Prophet Ibrahim (PBUH) and fear Allah (swt), believing in Prophet Muhammad (saws). Allah (swt) says:

 ( 127 ) وَإِذْ يَرْفَعُ إِبْرَاهِيمُ الْقَوَاعِدَ مِنَ الْبَيْتِ وَإِسْمَاعِيلُ رَبَّنَا تَقَبَّلْ مِنَّا ۖ إِنَّكَ أَنتَ السَّمِيعُ الْعَلِيمُ  ( 128 ) رَبَّنَا وَاجْعَلْنَا مُسْلِمَيْنِ لَكَ وَمِن ذُرِّيَّتِنَا أُمَّةً مُّسْلِمَةً لَّكَ وَأَرِنَا مَنَاسِكَنَا وَتُبْ عَلَيْنَا ۖ إِنَّكَ أَنتَ التَّوَّابُ الرَّحِيمُ ( 129 ) رَبَّنَا وَابْعَثْ فِيهِمْ رَسُولًا مِّنْهُمْ يَتْلُو عَلَيْهِمْ آيَاتِكَ وَيُعَلِّمُهُمُ الْكِتَابَ وَالْحِكْمَةَ وَيُزَكِّيهِمْ ۚ إِنَّكَ أَنتَ الْعَزِيزُ الْحَكِيمُ

(( 127 ) And [mention] when Abraham was raising the foundations of the House and [with him] Ishmael, [saying], “Our Lord, accept [this] from us. Indeed You are the Hearing, the Knowing. ( 128 ) Our Lord, and make us Muslims [in submission] to You and from our descendants a Muslim nation [in submission] to You. And show us our rites and accept our repentance. Indeed, You are the Accepting of repentance, the Merciful. ( 129 ) Our Lord, and send among them a messenger from themselves who will recite to them Your verses and teach them the Book and wisdom and purify them. Indeed, You are the Exalted in Might, the Wise.”) (Al-Baqara)

This is the invocation of Prophet Ibrahim (PBUH) when he built the Sacred House. He prayed for Prophet Muhammad (saws) and those who believe in Allah (swt) and have faith in Prophet Muhammad (saws). Those are the true custodians of the Sacred House, not the polytheists.