I'm also facing the same problem here. I had two Noise channels in my project and it was working properly but then i added another Noise channel now i'm facing the same issue mentioned above.
Thanks for sharing these details. Our team is looking into this, as typically NOISE in GarageBand should be able to run several more instances.
If anyone else is experiencing this behavior please let us know, and in particular feel free to share your system details with us.
A few questions will help us try to reproduce this:
Which Garageband and NOISE versions are you running?
Do any other tracks exist in the project?
Which presets in particular cause the memory to max out?