He is basically wanting to explore polygamy, and I am uncomfortable about it
You have to decide if you are okay with a non-monogamous relationship. It does come with its own set of experiences and challenges. If your answer is no to it, say so to him. If that is not acceptable to him, the two of you are not compatible.