How to properly set up ball physics in Unity?

Question

Unity

Unity has this scene in 3D: A

pink ball moves at a constant speed, ricochets when it hits a wall and obstacles.

Here are the settings of the RigidBody component of the ball

Box Collider hangs on walls and obstacles, there is no RigidBody.
Below I attach the code that hangs on the ball and is responsible for the movement of the ball and the ricochet.

public float startSpeed;
public float increasedStep;
public float maxSpeed;

private void FixedUpdate()
    {
            rb.velocity = direction * startSpeed;
     }

private void OnCollisionEnter(Collision other)
    {
        ContactPoint contactPoint = other.contacts[0];
        direction = Vector3.Reflect(direction, contactPoint.normal);

        if(startSpeed + increasedStep > maxSpeed) return;
        startSpeed += increasedStep;
    }

The problem is this: the ball does not always move at a constant speed, this can also be seen in the speed in the Info section of the ball's RigidBody component. In some situations, the speed of the ball changes (almost always decreases), for example when the ball moves close and parallel next to a wall or when it collides right into the corner of an obstacle.

Question: how to get the correct stable physical behavior of the ball?

Additionally: I tried not to use physics for movement, however, as far as I know, this practice should not be used when you need to count collisions with other objects, especially often. The result was worse. The ricochet trajectories are often not correct, sometimes the ball moves in a circle for example.

Answer 1

Well, there is such a thing as PhysicMaterial. Which is just responsible for friction, elasticity and so on.
But if you have a constant speed, then I would not really use physics for movement, but only detect collisions with it.