As commentators note, there are many malloc implementations, and the algorithm can vary significantly for each of these implementations. This is a vast and complex area, and you should read the memory management guide to get a complete idea on this topic.
In simple words, all malloc implementations are backed up using kernel memory management schemes. The kernel sees all system memory as pages of a fixed size (4k, 8k, etc.), and all allocations and deallocations are performed on the pages. There will be a memory management subsystem for all kernel implementations and which will take into account the allocations and releases of all memory in the system. When you call malloc, it will eventually reach this memory management subsystem and look for the next free page from the pool and allocate for the request process. Before passing the page to the requester, he will mark it as used and in the same way, when you free up memory, it will add it back to the free pool and remove the mark. There are so many implementations about how the kernel does it all effectively (read in Linux memory manager implementations)
In ordinary implementations, there is minimal functionality for the memory manager in user space itself. The user space process itself maintains a free pool, and when malloc requests memory before entering the kernel, it will search in its own free pool if memory is available. If available, it will mark it and satisfy the request without the help of the kernel. Similarly, when you free up memory, the freed memory chunk will not immediately return to the free kernel pool, but will remain in the free process pool so that the next malloc can use it.
As I said at the beginning, this is a huge and complex topic, and you can find many documents available on the Internet about this.