Wicca, a short story to calm you:
We have homes in two different places, over 500 kilometers (about 330 miles) apart. At both homes, we have the same internet service provider, ISP. In fact, both connections are on the same contract. Billing address is the second home, as we call it.
When one of the routers, here at home in Leipzig, or at second home in Solingen is connected to internet, it gets a so called external IP from a pool of IP's this service provider has assigned to both of these regions. Because the company is located in the west Germany, where the second home is, we always get IP's pointing to that area. For instance I'm now in Leipzig, but if you check my IP's location, it tells I'm close to Bielefeld that is about 450 kilometers, 290 miles away. In fact, our second home is much closer to that IP location.
So, checking the location of an IP does not show the exact place, address of that IP. It shows where this IP address is registered. My ISP has a big data center in Bielefeld, that's why "my location" is also there. Checking my IP only reveals I'm in Germany, not exactly where.
It's the same thing with you. Each time you get a new external IP, it points to a location of your ISP, not to your home. To find your home address, I would first need to contact your ISP with an IP address and the time and date when it was used. They then have registers to check which client had which IP at any certain time. Of course your ISP is not giving this information to anyone who asks, so you are quite safe.