The thing you linked is a modem-router, so it connects to the internet AND routes traffic between all the computers connected to it. Good.
Looking at the ports behind, it has only ONE Ethernet port. Bad, and where your issue is.
If you want to go 100% wired, you need an ethernet switch. It's a device that is smart enough to route the ethernet traffic from more than one PC through a single cable, but dumb enough that is plug and play.
Something like this
is cheap and tiny (i've seen tons of exactly the ones I linked and their bigger 24-port collegues in all companies with decent internal networks). Plug the ethernet cables from your router and from the computers into any of its 5 ports, as it doesn't matter, and you are ready to go.
If you want to do something fancy like connecting ethernet (network cable) devices to your wireless modem router through wireless connection without buying wireless adapters, look for a Wireless Ethernet Bridge
. These do need some setup to be integrated in your wireless network (as someone has to tell them the name of the wireless network they need to connect and the password), but it's usually straightforward as they are much simpler devices than the modem-router you have already.