Is the rca cable to jack only one way? So they can’t both be used for sending audio from computer and sending audio to computer?
Yes. The headphone jack on your macbook passes audio just one direction (out). When you use analog inputs or outputs your computer won't recognize the connected gadget(s) as a device. There is no "digital handshake" of ones and zeros like computer peripherals would use to identify themselves to one another. Analog inputs and outputs are just waiting to pass continuously varying voltages (in one direction) to the next device.
There are some macs with built in analog audio inputs, but you would know if you had one. It would have two different jacks on it.
Some macs are also equipped to send
digital audio out of their headphone jack using a variant of a fiber optic connection called Toslink with an 1/8" (3.5mm) Toslink adaptor. Some can also receive digital Toslink audio input (on their separate input jack) using the same type of Toslink adaptor. Even then, although Toslink IS a digital format, it needs separate connections and separate cables for input and output. [
="https://support.apple.com/guide/mac-help/about-audio-ports-cpmh0052/mac"]Further reading about mac audio ins and outs[/
].
As other have suggested, the best solution to do what you're describing with one microphone would be some sort of USB audio interface like this:
Scarlett. (or whatever number of mic inputs you'd like) Something like that would simultaneously send and receive audio over a single USB cable connected to your laptop. You can listen to what you're doing either by using headphones, or by connecting the Scarlett to a suitable speaker system.
Instead of the workflow I think you're envisioning; Playing music out of your laptop -> Into the Yamaha >< mixing the music and microphone together with the Yamaha and then -> sending that mix back into your laptop via some magic connection ... The more efficient method would be to connect the Scarlett (or similar device) to your computer > import the music file directly into the software included with the Scarlett >> plug the mic directly into the interface and avoid using the Yamaha mixer completely.
If you record the first way, [through the little Yamaha mixer], you are committed to that mix forever. You cannot change the mix afterward. If you have a great vocal performance, but don't get a good balance between the music and the vocal you are out of luck. Then you have to decide if you want to live with the bad mix, or keep on recording and hope you can recreate the great vocal take. You cannot isolate either source afterward and if you decide to add more vocal or instrument parts afterward, you would have no way to remix things to allow for the new parts.
Using the Scarlett (or similar device) with whatever software it includes, you can mix and edit everything using the software. Using the Scarlett software, you can mix it over and over again until you're happy with it and even save different variations along the way. Using the Scarlett, if you like all of the vocal performance except the ending, you can just "punch in" and retry the ending as many times as necessary. If you are inspired to add more parts later, simply record them alongside the original tracks and mix them all together when you're done. You can't do ANY of those things using the Yamaha as your mixer.
I hope that helps. Good luck!