How to change open files limit in Ubuntu 18.04?

Question

linux

When testing my application, I get an error like this

Горутина отработала за 3.213085ms
open /dev/null: too many open files
Горутина отработала за 3.258772ms
open /dev/null: too many open files
Горутина отработала за 3.196158ms
open ./flowscript.sh: too many open files
open ./flowscript.sh: too many open files
open /dev/null: too many open files
Горутина отработала за 3.570144ms

I found information that it is necessary to change the limit of open files in the OS, I tried it in different ways.

sudo sh -c "ulimit -Hn 500000 ; exec su \"$USER\""
 vi /etc/sysctl.conf
fs.file-max = 500000
# vi /etc/security/limits.conf
* soft nofile 60000
* hard nofile 60000

But still this error comes up.
Here is the code that I am calling

func exec_cmd(cmd string,nfFile string,wg *sync.WaitGroup,f func ()string,args ...string){

  t0 := time.Now()

  os.Mkdir("tmp",os.ModePerm)

  file, err := os.Create(bash_script);utils.Checkerr(err)
  defer file.Close()

  file.WriteString(binbash + "\n")
  file.WriteString(cmd)

  err = os.Chmod(bash_script,0777);utils.Checkerr(err)
  
  bashscript := exec.Command("bash",bash_script,nfFile,args[0],args[1],f());utils.Checkerr(err)
  err = bashscript.Run();utils.Checkerr(err)
  t1 := time.Now()
  fmt.Printf("Горутина отработала за %v\n", t1.Sub(t0))

  wg.Done()

}
func main(){

  t0 := time.Now()

  var path 		= flag.String("p", "","Path to directory (including month)")
  var filter 		= flag.String("f", "","Access list filename. Defaults to flow.acl.")
  var nf 			= flag.String("nf","","Select the active definition. Defaults to default")
  //var staticFile 	= flag.String("w", "","Write statistics to file")


  flag.Parse();
  flag.Usage = Usage

  if len(os.Args) == 1 {
    Usage()
    os.Exit(1)
  }

  name := TempFileName() // имена временных файлов

  var wg sync.WaitGroup
  NetflowFileNames := getAbsolutePathToFileNetflow(*path) // имена файлов за вчерашний день
  wg.Add(len(NetflowFileNames)) // 288
  for _, filename := range NetflowFileNames{
    go exec_cmd(cmdCommand,filename,&wg,name,*filter,*nf)

  }
  wg.Wait()
  os.RemoveAll(bash_script)
  t1 := time.Now()
  fmt.Printf("Время работы программы: %v\n", t1.Sub(t0))

}

and here I have a question, is it better to use goroutines with channels or the sync.WaitGroup type

Answer 1

Use a semaphore channel to control how many goroutines are running.

Answer 2

To make it work after a reboot, you can change it here: /etc/security/limits.conf
Note that it only takes effect in the next session.
But you don’t need to start with this)
1. You need to decide how many open descriptors you need.
A command that will show how much a certain process has opened:
lsof -p 9189 | wc -l
2. Be honest, do you really need that much? Or is it possible to optimize something in the code?
3. If you still need it, then yes, change hard and soft in /etc/security/limits.conf
4. Reboot or restart the session, if it doesn’t help, see step 1