How to correctly execute http.NewRequest with Cyrillic?

Question

go

Function Example

func response(ctx context.Context, url string, obj ResponseParser) error {
  req, err := http.NewRequest(http.MethodGet, url, nil)
  if err != nil {
    return err
  }
  req = req.WithContext(ctx)
  resp, err := http.DefaultClient.Do(req)
  if err != nil {
    return err
  }
  defer resp.Body.Close()
  data, err := ioutil.ReadAll(resp.Body)
  if err != nil {
    return err
  }
  body := strings.Trim(string(data), " []")
  if body == "" {
    return nil
  }
  if err := json.Unmarshal([]byte(body), obj); err != nil {
    return err
  }
  return nil
}

In the example, I'm trying to get data using the generated link ` url ` , if there is Cyrillic, then the result of the request is 400. If I paste the url into the browser, I get the necessary data. What could be the problem?

Answer 1

1. The case when the request contains Cyrillic characters:
The author had a url like:
The query string is part of the url and by standard cannot contain non-ASCII characters (RFC 1738). Since the url contains Cyrillic characters, the url needs to be encoded (URL encoding), which is what the QueryEscape function from the net/url package does.

package main

import (
  "fmt"
  "net/url"
)

func main() {
  q := "Запрос+На+Кириллице"
  u := "https://domain.name/data.json"
  fmt.Println(u + "?searchtext=" + url.QueryEscape(q))
}

$ go run main.go 
https://domain.name/data.json?searchtext=%D0%97%D0%B0%D0%BF%D1%80%D0%BE%D1%81%2B%D0%9D%D0%B0%2B%D0%9A%D0%B8%D1%80%D0%B8%D0%BB%D0%BB%D0%B8%D1%86%D0%B5

package main

import (
  "fmt"
  "golang.org/x/net/idna"
)
var p *idna.Profile

func main() {
  p = idna.New()
  fmt.Println(p.ToASCII("россия.рф"))
}

$ go run main.go 
xn--h1alffa9f.xn--p1ai <nil>

https://ru.wikipedia.org/wiki/IDN
https://ru.wikipedia.org/wiki/Punycode