First, reasonably, the system's cryptographic security - its unpredictability -, is not going to be altered in any way, regardless of whether the CPU's built-in HWRNG (e.g. RDRAND in Intel) is used or not, otherwise, as you noted, that would unavoidably weaken the security of the RNG (and anything that depends on it).
Briefly, just to sum up, during the bootstrapping process, after the kernel has loaded the randomness driver, the Linux RNG initializes all the random pools (entropy pools, which are just memory areas that hold random data), including the main pool (input pool), by populating them with entropy which comes from the HWRNG if it is available, otherwise via random_get_entropy(), which is a macro for get_cycles(), whose implementation varies with the architecture (e.g. in aarch64 a read of the CNTVCT_EL0 register is done, which is kind of a frequency counter, not actually the clock rate which is instead used in x86-64 by reading the TSC reg). All this data is given to a primary cipher state (primary_crng, an object of type struct crng_state, which is the implementation of the ChaCha20 algorithm), which contains a key of 384 truly random bits, which are eventually supplied to the /dev/urandom interface.
Now with this contextualization, to answer your question, kernel's RNG actual initialization through rand_initialize(), being an early_initcall, happens obviously only at boot time (like all the *_initcall()), particularly, at the very end of start_kernel(), the kernel routine rest_init() gets called where, one of the first things it does, is to spawn a kernel thread (kernel_init) with the purpose, among other things, to launch that initcall (note also that add_device_randomness() is called even before, but doesn't actually add any entropic data); and consequently yes, per source, the cipher state of the ChaCha20, the primary and the blocking pools would be initialized with arch_get_random_long(), which makes use of the RDRAND instruction in x86 processors (again, if it's available, and if it is, the kernel warns you about this). I wouldn't say it's the only source used though (even if RDRAND is available), because, at least:
The firmware time is used and is mixed into the pools during the initialization (maybe not that entropic, still an attacker would have to infer the timestamp with an extremely high precision);
There's another little pool (one per CPU, called fast pool), which collects entropy from add_interrupt_randomness(), which uses IRQs (and theoretically other kernel events, and even some seed coming from the CPU hardware RNG if there is) as input, all mixed, and then it's injected into the input pool. This happens like every second.
So the process of gathering entropy from both the HWRNG and the other sources happens all together; of course the first one dominates the scene (because the entropy quality is definitely higher, it's a true RNG), and occurs at boot time, and, from userspace, whenever the pseudorandom devices (or the getrandom() syscall) are used. In the second case though, the pools are already initialized, and just the primary_crng (the cipher state) is reseeded. And yes, as you noted, the CPU built-in RNG surely improves the overall randomness.
I would finally add, that, especially in the embedded systems, which may lack of some source noise (like keyboards, hard disks, etc.), the usage of the HWRNG would have a significant impact. And just note that, always per documentation, you could disable the kernel usage of RDRAND way before the introduction of RANDOM_TRUST_CPU, if you're concerned with security.
