How to eager load ONLY an array of ids from a link table?

Question

Yii

There are Job and JobCategory models. Multiple JobCategory can be associated with one Job. Their connection is implemented through a link table, and the method in the Job class looks like this:

public function getCategories() {
    return $this->hasMany(JobCategory::className(), ['id' => 'category_id'])
      ->viaTable(JobToCategoryLink::tableName(), ['job_id' => 'id']);
  }

This way I can eager load all Job-related categories:

Job::find()
  ->joinWith(['categories'])

However, this incurs a memory cost, because for each Job an array of JobCategory is created, each element of the JobCategory is a unique object, although many of the elements represent the same entry in the database.
Since I don’t have many JobCategory records, I cache them in memory in redis, wrote a JobCategoryRepository repository, which has the fetchAll($idList) method and returns JobCategory instances from the list of identifiers.
Question: how to eager load ONLY an ARRAY of CATEGORY IDs associated with a Job? In theory, I could make a query like:

public function getCategoryIdList() {
    return $this->hasMany(JobToCategoryLink::tableName(), ['job_id' => 'id'])
      ->addSelect(['category_id']);
  }

and load it via ->joinWith(['categoryIdList']), but in this case the categoryIdList relation is always an empty array. I can’t just return a Query with the required query parameters, because requests from relationals are executed either through ->one or ->all if the request object has multiple = true. And I just need ->column.
Any ideas?